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A  DECISION  SUPPORT  PROCEDURE  FOR 
BEST  VALUE  SOURCE  SELEQIONS 

Mkhael  F.  O'Cannoff  Janine  L  FariSf  and  Joan  5,  lovelace 


Here  we  discuss  the  use  of  decision  analytic  procedures  in  Best  Value  source 
selections.  Such  source  selections  specifically  require  that  choosing  an  offeror 
other  than  the  one  with  the  lowest  assessed  cost  must  be  justified  by  showing 
that  the  added  value  of  that  offeror's  proposal  is  worth  the  associated  extra 
cost  to  the  government.  Such  a  demonstration  implies  cost-benefit  tradeoffs 
with  an  associated  issue  of  benefit  quantification  during  the  source  selection. 
This  article  is  written  for  two  audiences.  One  is  acquisition  practitioners  who 
are  implementing  an  acquisition  program  that  will  employ  a  Best  Value  source 
selection.  We  aim  to  familiarize  this  audience  with  relevant  decision  analytic 
tradeoff  procedures  and  with  important  methodological  problems.  This  article 
will  familiarize  the  second  audience,  decision  analysis  practitioners,  with  an 
important  problem  for  which  their  tools  are  highly  relevant.  We  will  describe 
the  Best  Value  process,  provide  an  illustrative  procedural  template,  and  discuss 
methods  for  required  cost-benefit  tradeoffs.  The  report  also  addresses  legal 
issues  in  Best  Value  source  selections.  Finally,  the  report  presents  lessons 
learned  based  on  the  authors'  experience  in  Best  Value  acquisitions. 


Source  selection  is  the  formal  process 
by  which  the  government  makes  pro¬ 
curement  decisions  for  acquisitions. 
At  different  stages  of  an  acquisition, 
sources  (contractors)  are  chosen  to  de¬ 
velop  concepts,  to  conduct  studies,  to  de¬ 
velop  systems,  to  produce  a  system,  or  to 
provide  services.  Decision  issues  are  not 


separated  by  the  artificial  barriers  created 
by  the  sequenced  programmatic  approach, 
but  instead  transcend  those  boundaries. 
Decision  support  procedures  and  tools 
should  help  link  each  decision  to  the  ulti¬ 
mate  benefits  and  costs  associated  with  the 
eventual  decision  outcomes. 


The  view,  opinions,  and  findings  in  this  report  are  those  of  The  MITRE  Corporation  and  shouid 
not  be  construed  as  an  officiai  government  position,  poiicy,  or  decision,  uniess  designated  by 
other  documentation. 
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THE  PROCESS 


In  the  Best  Value  approach  to  source 
selection,  the  Source  Selection  Evaluation 
Board  (SSEB)  makes  its  report  to  the 
Source  Selection  Advisory  Council 
(SS  AC),  designating  evaluation  results  for 
those  offeror  proposals  in  the  competitive 
range.  The  SSAC,  unlike  the  SSEB,  can 
actually  compare  the  offeror  proposals 
against  each  other.  The  award  is  then  made 
to  the  “most  advantageous  alternative  to 
the  government.”  The  Best  Value  compari¬ 
son  process  is  triggered  if  that  proposal 
with  the  lowest  total  co.st  is  not  also  con¬ 
sidered  to  offer  the  greatest  benefit  to  the 
government  (Federal  Contracts  Report, 
1993;  Sochon.  1994;  USGSA,  1992).  It  must 
be  noted  that  the  so-called  Best  Value  ap¬ 


proach  does  not  implement  anything  that 
was  not  already  permissible  under  the 
Federal  Acquisition  Regulation  (FAR, 
1993).  However  the  strong  precedent  for 
competition  and  choosing  the  lowest  bid¬ 
ders,  and  the  fear  of  the  tremendous  costs 
and  associated  damage  caused  by  the  pro¬ 
gram  delays  that  occur  with  protests,  re¬ 
quire  clarification  of  this  procedure.  The 
Best  Value  variation  of  source  selection 
generally  proceeds  as  follows; 

•  The  SSAC  members  (or  a  special  Best 
Value  team  assisting  the  SSAC)  com¬ 
pare  the  offeror  proposals  (not  the  pro¬ 
posal  scores)  based  on  the  information 
provided  them  in  the  report  and  brief¬ 
ings  by  the  SSEB. 


Michael  F.  O'Connor,  Ph.D.,  is  a  lead  scientist  in  the  Economic  and  Decision  Analysis  Center 
of  the  MITRE  Corporation  specializing  in  the  application  of  decision  analytic  techniques  to 
complex  organizational  decisions  for  DoD  and  other  government  organizations.  He  has  ap¬ 
plied  these  techniques  for  more  than  20  years  to  program  design  decisions,  system  design 
and  acquisition  decisions,  budgeting  decisions,  policy  decisions,  and  planning  exercises.  He 
has  recently  focused  on  Best  Value  source  selection  decisions.  He  received  a  Ph.D.  in  math¬ 
ematical  psychology  from  the  University  of  Michigan.  He  has  taught  at  several  universities 
including  the  University  of  Southern  California,  the  University  of  Chicago,  George  Mason  Uni¬ 
versity,  and  the  George  Washington  University. 

Janine  L.  Paris  is  a  principal  member  of  the  technical  staff  in  the  Economic  and  Decision 
Analysis  Center  of  The  MITRE  Corporation  in  McLean,  VA.  She  is  responsible  for  product  qual¬ 
ity,  program  development,  staff  development,  and  technical  support  in  the  delivery  of  consult¬ 
ing  services  for  acquisition  support  and  economic  analysis  to  multipie  projects  for  high  tech¬ 
nology  programs  of  the  federal  government.  Recent  clients  include  the  Defense  Information 
Systems  Agency  (DISA)  and  the  Director  of  Defense  Research  and  Engineering  (DDR&E), 
Paris's  previous  position  at  MITRE  was  as  a  group  leader  for  the  Resource  Analysis  Group.  She 
holds  an  M  B  A.  degree  from  Loyola  College,  a  B.A.  degree  from  Antioch  College,  and  a  Certifi¬ 
cate  in  Procurement  and  Contracts  Management  from  the  University  of  Virginia.  Paris  is  a  cer¬ 
tified  cost  estimator/analyst  through  the  Society  of  Cost  Estimating  and  Analysis  (SCEA). 

Joan  S.  Lovelace  is  currently  a  department  head  for  the  Business  and  Economic  Analysis 
Center  in  Mitretek  Systems,  Inc.  She  is  a  graduate  of  the  Defense  Systems  Management 
College's  Program  Management  Course.  She  has  served  as  the  National  Conference  Program 
Chair  for  the  Society  of  Cost  Estimating  and  Analysis,  and  is  currently  president  of  the  Wash¬ 
ington  Chapter  of  SCEA.  She  is  a  member  of  the  International  Society  of  Parametric  Analysts 
and  is  a  former  editor  of  the  Journal  of  Parametrics.  She  holds  a  B.A.  degree  in  American 
studies  from  Beloit  College,  an  M.A.  degree  in  history  from  Claremont  Graduate  School,  and 
an  M.S.L.S.  from  Catholic  University. 
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•  Discriminators  (i.e.,  relative  advan¬ 
tages  that  one  proposal  has  over  an¬ 
other)  among  the  proposals  are  estab¬ 
lished  through  this  comparative  pro¬ 
cess.  Discriminators  will  relate  to  one 
or  more  of  the  evaluation  factors  used 
in  the  SSEB  evaluation  process  and 
must  be  traceable  to  those  factor  evalu¬ 
ations. 

•  Impact  areas  of  the  discriminators  are 
defined — that  is,  the  benefit  areas  af¬ 
fected  by  the  discriminators.  A  dis¬ 
criminator  might  be  some  aspect  of  the 
offeror’s  technical  approach  to  an  im¬ 
portant  system  capability.  The  associ¬ 
ated  impact  area  would  be  some  aspect 
of  conditional  expected  performance. 
(Note  that  benefits  are  assigned  to  im¬ 
pact  areas,  not  to  the  discriminator  dif¬ 
ferences.)  The  benefit  assessment  is 
thus  a  prediction,  and  assessment  pro¬ 
cedures  must  consider  the  uncertainty 
involved. 

•  Discriminator  impact  areas  are  often 
characterized  in  Best  Value  methodol¬ 
ogy  descriptions  as  either  “quantified” 
or  “nonquantified.”  The  term  “quan¬ 
tify”  is  ambiguous.  Procedurally,  in 
source  selections,  it  usually  means  to 
assign  dollar  values  to  levels  of  the 
variable  to  be  quantified.  To  engineers 
it  seems  to  mean  to  assess  on  an  inter¬ 
val  or  ratio  scale.  For  example,  a  quan¬ 
tified  discriminator  impact  area  might 
be  “expected  payload.”  A  non-quanti- 
fied  (sometimes  called  “qualified”)  dis¬ 
criminator  impact  area  might  be  “ex¬ 
pected  program  management  effi¬ 
ciency.”  Such  distinctions  are  not  very 
useful  from  a  measurement  theoretic 


point  of  view,  but  can  have  strong  im¬ 
plications  as  used  in  the  currently  de¬ 
fined  acquisition  process.  Furthermore, 
as  indicated,  these  benefit  impacts  are 
uncertain,  and  the  uncertainty  must  be 
characterized  (even  if  it  can’t  be  di¬ 
rectly  quantified). 

•  The  benefits  associated  with  the  impact 
of  each  discriminator  are  described  us¬ 
ing  as  much  clarity  of  definition  and 
precision  of  assessment  as  is  feasible. 
As  indicated,  benefits  are  assessed  for 
predicted  impacts,  and  the  expectation 
and  uncertainty  should  be  assessed. 

•  The  benefits  associated  with  quantified 
discriminators  are  traded  off  against  the 
total  cost  to  the  government  (not  sim¬ 
ply  contract  price).  These  total  cost  es¬ 
timates  should  be  adjusted  for  cost  risk 
(also  characterized  as  cost  realism).  As 
indicated,  the  described  tradeoff  trans¬ 
lates  benefits  to  dollars.  Less  stringent 
tradeoff  requirements  allow  for  system¬ 
atic  judgmental  assessments  of  the 
worth  of  the  benefits  relative  to  each 
other  and  relative  to  cost.  The  tradeoff 
process  depends  on  the  implementation 
method  chosen.  At  this  point,  a  single, 
well-defined  requirement  for  the  sca¬ 
lar  aspects  of  the  tradeoff  process  has 
not  been  defined.  Several  alternatives 
are  discussed  in  the  next  section. 

•  Nonquantified  benefits  associated  with 
discriminators  are  also  used.  Although 
these  benefits  are  used  only  as  tie¬ 
breakers  by  some  agencies,  they  can 
be  traded  off  against  other  benefits  or 
costs,  which  is  the  procedure  recom¬ 
mended  here. 
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•  The  resulting  implications  are  exam¬ 
ined  by  the  SS  AC  and  a  decision  is  rec¬ 
ommended  to  the  Source  Selection  Au¬ 
thority  (SSA). 

Varutions  of  the  Approach 

Several  methods  for  trading  off  cost  and 
benefits  in  Best  Value  decisions  have  been 
described.  Four  are  reported  here,  but  for 
the  reasons  explained  below,  only  meth¬ 
ods  2  and  3  are  recommended. 

1 .  Dollar  cost  to  obtain  increa.sed  benefit. 
In  this  method  an  estimate  is  estab¬ 
lished  of  the  cost  to  provide  from  the 
lower  cost,  lower  rated  technical  pro¬ 
posal  a  benefit  of  comparable  magni¬ 
tude  to  that  associated  with  the  higher 
rated  technical  proposal.  How  could 
the  lower  cost  proposal  be  be.st  up¬ 
graded  to  achieve  the  key  benefits  (as 
indicated  by  discriminator  impact  ar¬ 
eas)  of  the  higher  rated  technical  pro¬ 
posal,  and  what  would  be  the  cost  to 
do  so?  This  adjustment  should  be  in 
terms  of  the  most  probable  total  cost 
to  the  government  and  should  adjust 
for  cost  realism  and  fordi.scounted  ca.sh 
flow.  The  decision  would  then  be  made 
in  terms  of  the  adjusted  cost  figures. 
The  idea  seems  simple.  If  the  cost  to 
provide  the  added  benefit  is  less  than 
the  actual  cost  difference,  then  the 
added  benefit  of  the  higher  rated  tech¬ 
nical  proposal  is  not  worth  its  added 
cost.  Thus  the  lower  bid  provides  the 
greatest  value  in  a  cost-benefit  sen.se. 
This  procedure  does  not  suggest  “tech¬ 
nical  leveling”  (i.e..  to  actually  tran.s- 
mit  the  ideas  to  the  lower  bidder).  It  is 
simply  a  cost  estimating  methodology 
designed  to  answer  the  Best  Value 
question. 


A  potentially  serious  problem  with  this 
procedure  is  that  the  technical  approaches 
of  the  offerors  may  be  so  different  that  the 
cost  to  change  the  lower  cost,  lower  rated 
technical  proposal  to  provide  the  techni¬ 
cal  benefit  of  the  higher  cost,  higher  rated 
proposal  is  prohibitively  large,  or  cannot 
be  reasonably  calculated,  indicating 
choice  of  the  higher  rated  proposal.  Yet, 
the  government  may  assess  that  the  per¬ 
ceived  value  of  the  benefit  associated  with 
this  technical  superiority  is  minimal,  less 
than  the  cost  difference  and  it  clearly 
would  not  pay  this  difference.  This  indi¬ 
cates  that  the  lower  bidder  should  be  cho¬ 
sen.  It  also  shows  that  this  tradeoff  method 
can  produce  different  results  from  the  fol¬ 
lowing  two  methods.  The  cost  to  provide 
a  benefit  is  not  necessarily  a  valid  indica¬ 
tion  of  the  value  of  the  benefit  to  the  deci¬ 
sion  maker  as  reflected  by  “willingness 
to  pay”  (see  Discriminator  Benefit 
Tradeoff  method).  This  method  conse¬ 
quently  is  not  usually  a  desirable  approach 
from  a  decision  analytic  point  of  view;  we 
do  not  recommend  it  here. 

2.  Direct  discriminator  impacts  benefits 
quantification.  The  direct  di.scrimina- 
tor  impacts  benefits  quantification  in¬ 
volves  directly  assessing  the  value  to 
the  government  of  the  effects  associ¬ 
ated  with  discriminator  advantages  of 
each  offeror.  For  each  discriminator, 
the  impact  area  is  identified  and  an  at¬ 
tempt  is  made  to  directly  quantify  the 
benefit  of  the  impact  using  a  model, 
an  established  methodology,  a  surro¬ 
gate  index,  or  some  other  measure  that 
characterizes  the  benefit  of  the  impact. 
An  example  would  be  increased  pro¬ 
ductivity  of  a  workforce  attributable  to 
increased  user  friendliness  of  computer 


138 


A  Decision  Support  Procedure  hr  Best  Value  Source  Selections 


workstations.  A  discriminator  can  be 
unique  to  a  particular  offeror,  or  sev¬ 
eral  offerors’  proposals  may  have  po¬ 
tential  impacts  with  respect  to  a  dis¬ 
criminator.  All  important  discrimina¬ 
tors  are  quantified  where  possible. 

An  important  consideration  related  to 
this  and  the  procedure  described  is  that 
generally  in  offeror  comparisons,  each  of¬ 
feror  has  one  or  more  advantages  over 
other  offerors.  Thus  the  lowest  bidder  may 
also  have  several  benefit  advantages  over 
the  higher  bidder  even  though  the  prepon¬ 
derance  of  benefits  clearly  lies  with  the 
higher  bidder.  Once  discriminator  impacts 
have  been  quantified  for  all  offerors,  costs 
are  appropriately  adjusted  for  each  offer 
to  yield  adjusted  net  values.  (See  “A  Pro¬ 
cedural  Example”  for  more  information 
on  the  procedure  and  terminology.)  A  prac¬ 
tical  issue  is  that  some  discriminators  can¬ 
not  be  quantified,'  and  some  discrimina¬ 
tors  can  only  be  partially  quantified.  When 
this  occurs,  the  nonquantified  discrimina¬ 
tor  impacts  can  be  traded  off  against  quan¬ 
tified  ones  using  the  third  procedure  dis¬ 
cussed  here,  or  they  can  be  used  as  the 
breakers. 

3.  Discriminator  benefit  tradeoff.  In  this 
method  the  SSAC  determines  the  rela¬ 
tive  value  of  the  increased  benefits  as¬ 
sociated  with  the  technically  superior 
proposal  in  direct  tradeoffs,  in  which 
described  benefits  are  directly  com¬ 
pared  pairwise  using  ordinal  preference 
or  ranking  type  assessments.  “Willing¬ 
ness  to  pay”  is  often  used  as  a  prefer¬ 
ence  assessment  method.  All  attribute 
differences  or  advantage  impacts 
among  alternatives  become  “benefits” 
for  tradeoff  in  this  approach.  If  there 


are  more  than  two  offerors  in  the  Best 
Value  comparison,  as  is  usually  the 
case,  benefits  are  traded  off  for  each 
pair  of  offerors.  A  rank  ordering  of  the 
benefits  can  be  established  using  this 
process  which  can  also  provide  con¬ 
sistency  checks.  A  version  of  the  pro¬ 
cess  is  described  in  the  example  ap¬ 
proach  presented  in  the  next  section. 
Tools  such  as  Multi-Attribute  Utility 
Theory  (MAUT)  or  the  Analytic  Hier¬ 
archy  Process  (AHP)  can  be  used  in 
establishing  such  an  ordering.  These 
tools,  if  employed  using  appropriate 
axiomatic  tests,  can  provide  interval 
level  utility  assessments.  The  results  of 
the  discriminator  benefit  tradeoff 
method  are  likely  to  differ  from  those 
of  the  first  method  described.  At  the 
same  time,  the  ordering  of  options  de¬ 
rived  from  this  approach  should  agree 
fairly  well  with  those  of  the  second 
method,  and  should  provide  a  valid  rep¬ 
resentation  of  the  decision  maker’s 
values. 

4.  Point  scoring.  In  this  method,  price 
points  and  technical  points  are  assigned 
in  a  manner  established  in  the  Source 
Selection  Evaluation  Plan  (SSEP)  and 
described  in  the  Request  For  Propos¬ 
als  (RFP).  This  procedure  when  used 
by  the  SSEB  can  be  somewhat  restric¬ 
tive  and  is  usually  avoided  in  source 
selections  for  this  reason.  Assigning 
points  to  discriminator  advantages  pro¬ 
portionally  to  the  weights  of  evalua¬ 
tion  criteria  on  which  the  advantage  is 
based  may  not  validly  quantify  the 
value  of  the  resultant  benefits  associ¬ 
ated  with  discriminator  impact  areas, 
thus  misrepresenting  discriminator 
benefit.  These  relative  benefits  must 
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still  be  established  and  justified  as  a 
meaningful  difference  to  establish  Best 
Value,  for  as  indicated,  discriminators 
need  not  be  the  actual  evaluation  crite¬ 
ria  of  the  SSEP.  If  points  are  assigned 
that  actually  represent  the  value  of  the 
discriminator  impact  to  the  govern¬ 
ment,  this  accomplishes  the  same  re¬ 
sult  as  the  tradeoff  approach  of  meth¬ 
ods  2  or  3  above  (which  must  ,still  be 
used  to  assure  consistency),  and  the  ap¬ 
proach  is  simply  a  variation  of  one  of 
those.  Attempting  to  assure  that  the  re¬ 
sultant  benefit  assignments  do  not  vio¬ 
late  implications  of  the  point  di.stribu- 
tion  rules  set  forth  in  the  SSEP  and 
partially  explained  in  the  REP  is  very 
difficult,  especially  if  a  discriminator 
relates  to  more  than  one  evaluation  cri¬ 
terion.  The  ordering  of  the  impact  ben¬ 
efits  thus  derived  can  thus  potentially 
be  different  from  what  might  be  ex¬ 
pected  from  the  ordering  of  the  impor¬ 
tance  of  weights  assigned  in  the  REP. 
It’s  a  good  idea  to  examine  the  impli¬ 
cations  of  such  cases  as  a  check  on  the 
analysis,  but  that  should  be  the  extent 
of  the  procedure. 


A  Procedural  Example:  The  Best 
Value  Assessmeht  Pian  For 
SYsnM  X  Source  SELEaioN 


Here  we  give  an  example  of  the  gen¬ 
eral  approach  to  Best  Value  for  the  hypo¬ 
thetical  System  X  source  selection.  The 
approach  is  based  on  accepted  practice, 
the  legal  and  procedural  rulings  with  re¬ 
spect  to  Best  Value  procurements,  and  cer¬ 
tain  assumptions  about  quantification. 
This  approach  is  to  be  implemented  when 


the  SSEB  evaluation  has  been  completed 
and  the  issue  of  Best  Value  must  be  ad¬ 
dressed  by  the  SSAC.  Because  the  cru¬ 
cial  question  is  overall  value  to  the  gov¬ 
ernment,  certain  steps  related  to  cost  real¬ 
ism  and  ri.sk  are  included.  This  is  a  spe¬ 
cific  example,  and  variations  dependent 
on  procurement  conditions  are  quite  ap¬ 
propriate  (see  Faris  and  Lovelace,  1994, 
for  further  discussion).  It  also  tries  not  to 
do  violence  to  current  practice  while  still 
in.serting  accepted,  valid  assessment  tech¬ 
niques  into  the  process.  The  treatment  of 
uncertainty  as,sessments  and  cost-benefit 
tradeoffs  is  necessarily  abbreviated  and 
other  variations  using  benefit  and  cost 
ranges  are  acceptable. 

Recommended  Snps 

Step  1.  A  Best  Value  working  group 
(BVWG)  is  formed.  This  is  done  after 
SSEB  reports  are  finished.  This  BVWG 
should  be  small,  but  the  group  in  aggre¬ 
gate  should  represent  complete  knowledge 
of  proposal  cost,  management,  and  tech¬ 
nical  content  (as  well  as  any  .special  evalu¬ 
ation  areas,  e.g.,  software  capability  evalu¬ 
ation  [SCE]).  The  BVWG  should  serve  in 
an  advisory  role  to  the  SSAC.  BVWG 
members  should  be  familiar  with  the 
evaluations  done  by  the  SSEB  and  the 
summary  report  preparations,  and  they 
need  access  to  all  relevant  information. 
Initially  the  BVWG  can  be  formed  mostly 
of  tho,se  familiar  with  the  SSEB  evalua¬ 
tions  in  order  to  efficiently  define  discrimi¬ 
nators.  Then  a  subset  of  SSAC  members 
may  Join  to  enhance  the  identification  of 
impact  areas  and  a,ssociated  benefits. 

Step  2.  The  BVWG  reviews  the  cost 
realism  information.  The  BVWG  should 
review  cost  realism  work  done  and  pre- 
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pare  the  relevant  information  in  a  tabular 
format.  If  the  required  information  is  not 
available,  the  BVWG  should  prepare  the 
information  from  data  provided  by  the 
SSEB. 

This  cost  information  should  include  all 
adjustments  to  price  and  cost  to  reflect  an 
accurate  assessment  of  the  true  total  cost 
to  the  government  of  the  offeror’s  proposal 
(sometimes  denoted  as  the  “government’s 
most  probable  cost,”  or  GMPC).  Adjust¬ 
ments  made  to  the  offeror’s  cost  can  be  to 
correct  arithmetic  errors,  adjust  inappro¬ 
priate  labor  mixes,  or  to  reflect  technical 
or  management-related  risks  that  will  af¬ 
fect  the  effort  or  schedule  required  to  de¬ 
liver  the  capability  as  proposed.  (See 
Carroll,  1994,  for  further  discussion.) 

Where  schedule  adjustments  are  made, 
they  are  also  quantified  as  part  of  a  cost 
realism  assessment.  Both  schedule  and 
cost  risk  are  determined  by  the  major  un¬ 
certain  events  in  the  program  life  cycle, 
some  due  to  the  nature  of  the  offeror’s 
proposal  and  some  external  to  the  pro¬ 
posal.  Costs  are  conditional  on  schedule, 
and  uncertainty  assessments  must  accom¬ 
modate  such  dependencies.  One  must 
therefore  carefully  integrate  cost  and 
schedule  realism  assessments. 

Another  consideration  involves  the  use 
of  the  total  acquisition  cost  of  the  program 
to  the  government  as  a  second  cost  met¬ 
ric,  in  addition  to  the  life-cycle  cost,  for 
comparing  different  offeror  proposals. 
When  schedule  corrections  are  made,  any 
impact  on  the  government  effort  (includ¬ 
ing  support  personnel)  required  to  man¬ 
age  the  program  can  also  be  accounted  for 
and  included  in  an  adjusted  total  cost  num¬ 
ber.  This  assessment  can  be  difficult  for 
some  contracts  involving  large  support 
structures,  and  if  a  simplified  approach  is 


chosen  for  that  reason,  it  must  be  even- 
handed  with  respect  to  the  evaluations  of 
all  offers. 

Risks  external  to  the  offeror’s  proposal 
but  part  of  the  system  life  cycle  and  thus 
the  total  cost  to  the  government  are  often 
included  as  quantified  (or  qualified)  risks. 
Such  a  risk  could  be  due  to  some  event 
(e.g.,  a  technology  requirement  upon 
which  the  system  employment  is  depen¬ 
dent)  that  causes  uncertainty  unique  to  the 
offeror’s  ap¬ 
proach  to  the  .aufion  is  thal 

program,  and  risk  inclusion  should 
for  which  a  spe-  accomplished 

cific  program  evenhandediy  so 
cost  assessment  lhal  all  important 
cannot  be  rea-  risks  are  included 
sonably  made,  for  all  offerors 
This  partition-  during  the  same 
ing  of  risks  can  P**"*®**." 
be  a  matter  of 

preference.  The  impact  of  all  risks  that  can 
be  quantified  can  be  included  in  this  cost 
realism  adjustment,  both  program  inter¬ 
nal  and  program  external  risks.  The  de¬ 
sire  is  that  the  resulting  number  be  a  real¬ 
istic  estimate  of  the  most  probable  life- 
cycle  cost  to  the  government  of  choosing 
the  offeror  in  question  to  provide  the  de¬ 
sired  capability  or  system.  A  caution  is  that 
risk  inclusion  should  be  accomplished 
evenhandediy  so  that  all  important  risks 
are  included  for  all  offerors  during  the 
same  period. 

All  adjustments  to  proposed  cost  or 
price  should  be  recapped  in  tabular  for¬ 
mat  and  supported  by  descriptive  meth¬ 
odology  detailing  the  logic  and  providing 
an  audit  trail  (Table  1).  A  separate  but  re¬ 
lated  table  should  account  for  uncertainty 
in  the  adjustments  made  by  indicating  the 
uncertainty  ranges  associated  with  the  as- 
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Table  1. 

Cost  Realism  Table 


Cost  Realism  Assessment  (SM) 

Offeror  XI 

Offeror  X2 

Offeror  X3 

Offeror  X4 

1 .  Proposed  cost  (for  proposed  effort) 

$54.5 

$59.3 

$65.0 

$50.2 

2.  Arithmetic  cost  adjustment 
(for  errors,  omissions,  etc.) 

0.5 

0.2 

0.0 

(0.2) 

3.  Cost  adjustment  of  offeror  effort. 

This  adjustment  results  from  an 
independent  assessment  of 
technicaf-'management  effort 
required  to  deliver  the  system 
as  proposed  by  the  offeror. 

1.2 

0.7 

1.5 

3.0 

4.  Cost  adjustments  due  to 

technical  and/or  management  risks. 
These  will  impact  the  delivery 
schedule  or  effort  required  to 
deliver  the  proposed  system  as 
scheduled:  they  can  be  due  to 
overly  optimistic  schedule 
assumptions,  complex  technical 
interdependencies  not  accounted 
for  in  the  proposed  effort,  and  the  like. 

6.0 

2.5 

1.0 

8.0 

Total  -  cost  realism  assessment 

$62.2 

$62.7 

$67.5 

$61.0 

sessments.  An  appropriate  confidence  in- 
ten-al  (e.g.,  80  percent)  should  be  consis¬ 
tently  used  for  discussions  of  uncertainty. 
The  choice  of  an  80  percent  confidence 
inters'al  instead  of,  say,  a  95  percent  range, 
itself  reflects  a  tradeoff.  The  important 
point  is  that  cost  adjustments  are  uncer¬ 
tain  assessments,  and  the  magnitude  of  the 
uncertainty  should  be  indicated  to  deci¬ 
sion  makers. 

Step  3.  The  BVWG  works  with  the 
SSAC  to  establish  discriminators.  The 
BVWG  res'iews  the  technical,  manage¬ 
ment,  and  co.st  evaluations.  In  doing  so, 
the  BVWG  reviews  offeror  strengths  and 
weaknesses  with  respect  to  the  evaluation 
factors  as  summarized  by  the  SSEB  evalu¬ 
ation  final  reports.  The  BVWG  conducts 


a  comparative  evaluation  of  these  offeror 
strengths  and  weaknesses  to  identify  com¬ 
parative  advantages  and  weaknesses  as¬ 
sociated  with  each  offeror’s  proposal. 
These  advantages  and  weaknesses  iden¬ 
tify  discriminators  among  offeror  propos¬ 
als. 

Each  discriminator  is  traced  to  the 
evaluation  factor(s)  from  which  it  is  de¬ 
rived  and  these  discriminator/evaluation 
factor  links  are  listed.  The  impact  area  of 
each  discriminator  is  also  listed.  This  is 
the  potential  programmatic  or  operational 
benefit  (or  risk)  associated  with  the  dis¬ 
criminator.  For  each  discriminator  identi¬ 
fied  for  each  offeror,  the  impact(s)  are 
described  in  sufficient  detail  to  support  the 
resultant  benefits  assessment,  and  provide 
summary  rationale  for  the  benefit.  Ben- 
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efits  could  include  reduced  system  down 
time  or  enhanced  interoperability  with 
other  Department  of  Defense  (DoD)  sys¬ 
tems.  Technical  or  management  risks  that 
are  related  to  events  beyond  the  control 
of  the  offeror  and  that  thus  were  not  pre¬ 
viously  accounted  for  in  cost  realism  ad¬ 
justments  can  also  be  included  here  as  dis¬ 
criminators.  As  indicated  earlier,  when  dis¬ 
criminator  impacts  and  issues  of  quantifi¬ 
cation  are  addressed,  it  is  beneficial  to  in¬ 
volve  some  SSAC  members.  The  appro¬ 
priate  member  depends  on  several  factors 
relating  to  efficiency  and  validity  of  the 
process.  Using  the  benefit  description  de¬ 
veloped,  one  makes  a  determination  con¬ 
cerning  which  discriminator  impacts  can 
be  quantified  (e.g.,  as  a  potential  cost 
avoidance  in  development,  as  a  cost  sav¬ 
ings  in  program  management  and  thus 


program  costs,  as  a  cost  savings  in  opera¬ 
tions  and  support,  as  an  operational  ben¬ 
efit  that  can  be  partially  quantified,  as  a 
quality,  capability  enhancement). 
Nonquantifiable  benefits  and  risks  devel¬ 
oped  in  the  above  description  are  listed 
with  supporting  rationale.  These  can  be 
recapped  in  tabular  format  (Table  2). 

Step  4.  A  benefits  summary  table  is 
prepared.  This  table  relates  the  offeror 
benefit  checks  in  the  above  table  to  a  ben¬ 
efits  reference  summary,  which  should  de¬ 
scribe  in  detail  the  rationale  for  the  ben¬ 
efit  claimed  for  the  offeror.  This  latter  de¬ 
scription  should  be  systematic  and  should 
include  referenced  precedents,  studies, 
similar  systems,  and  any  other  relevant  in¬ 
formation  used  to  assess  the  benefit  asso¬ 
ciated  with  the  discriminator.  If  the  ben- 


Table  2. 


Discriminator  Impacts  Table 


Offerors 

Discriminator 

Impact  area 
of 

discriminator 

Evaluation  factor 
from  which 
discriminator  is 
derived  (can  be 
more  than  one). 

Quantitative? 
Indicate  if 
benefit  can 
be,  or  has 
been 

quantified. 

Qualitative? 
Indicate  if 
benefit  can¬ 
not  be 
quantified 
and  is 
supported 
only  by 
descriptive 
rationale. 

X 

1 

X 

2 

X 

3 

1.  Key 
personnei 

R  M,C 

Management 

X 

B 

-1- 

1 

1 

2.  Early 
delivery 

Mm 

Technical/ 

overall 

X 

+ 

-1- 

1 

1 

3.  Past 

performance 

P,  M 

Related 

experience 

X 

B 

B 

B 

+ 

4.  Training 

CL 

Management 

X 

-1- 

+ 

-1- 

*P  =  productivity,  M  =  mission  effectiveness,  C  =  cost.  No  entry  means  no  impact,  +  indicates  a  positive 
impact  for  the  offeror,  and  -  indicates  a  negative  impact  or  risk  for  the  offeror. 
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efit  will  be  quantified,  this  should  also  be 
indicated.  The  benefits  summary  table 
contains  a  ver>'  brief  summary  of  the  con¬ 
clusions  to  be  drawn. 

Step  5.  The  BVVVG  quantifies  the 
benefits  listed  as  quantifiable  in  the  dis¬ 
criminator  impacts  table  (Table  2). 

These  benefits  are  quantified  and  inserted 
into  the  appropriate  cells  for  appropriate 
offerors  in  the  quantified  benefits  table 
below.  The  methods  for  quantification  can 
include  analy.ses  specific  to  the  impact 
area.  u.se  of  a  particular  cost  or  related 
model  identified  early  in  the  source  selec¬ 
tion  process,  or  use  of  some  logically  de¬ 
veloped  algorithm.  Generally,  these  ben¬ 
efits  will  be  positive,  reflected  as  deltas 
above  the  offeror  scoring  lowest  overall 
on  the  evaluation  criterion  (or  criteria) 
linked  to  that  discriminator.  The  benefits 
summary'  table  or  a  similar  report  indicated 
in  Step  4,  explaining  the  basis  of  each  and 
eveiy  predicted  benefit,  is  prepared  and 
referenced.  The  explanation  includes  the 
logic  for  the  quantification  method  chosen, 
referenced  precedents,  studies,  information 
sources,  etc.  A  description  of  the  procedure 
should  be  provided  sufficient  to  facilitate  a 
clear  understanding  of  how  quantification 
of  the  benefit  was  accomplished. 


At  this  time,  the  BVWG  also  quanti¬ 
fies  those  technology  and  business  risks, 
identified  in  the  discriminator  impact 
table,  that  can  be  quantified.  As  indicated, 
these  may  be  due  to  potential  events  out¬ 
side  the  offeror’s  control  that  can  occur  at 
any  point  in  the  system  life  cycle,  or  they 
can  be  risks  that  have  not  yet  been  con¬ 
sidered  in  the  evaluation  process.  This 
means  that  the  risks  are  based  on  infor¬ 
mation  that  has  not  already  been  ac¬ 
counted  for  in  the  cost  realism  adjustment 
made  (Table  1)  described  in  Step  2.  Where 
possible,  and  if  appropriate,  the  potential 
impact  of  the  identified  risk  is  quantified 
as  a  cost  impact  on  the  program.  These 
most  probable  quantified  technical  or 
management  risks  are  also  inserted  into 
the  appropriate  cells  in  Table  3.  Generally 
such  risks  will  have  a  negative  value — 
that  is,  they  are  “negative  benefits”  or 
“disbenefits”  which  will  thus  increase  the 
overall  total  cost  to  the  government. 

The  impact  of  particular  technical  or 
business  risks  are  uncertain,  and  the 
BVWG  can  assess  the  range  of  potential 
cost  impacts  from  the  minimum  to  a  maxi¬ 
mum  figure  for  each  individual  risk.  As 
indicated,  the  most  probable  value  is  in¬ 
serted  in  the  table  as  a  point  estimate;  how¬ 
ever,  an  appropriate  cumulative  probable 


Table  3. 

Quantified  Benefits  Table  ($M) 


Discriminator  impact 
(Quantified  benefits  from  Table  2) 

Offeror  XI 

Offeror  X2 

Offeror  X3 

Offeror  X4 

Early  delivery 

$1.5 

$2.5 

- 

- 

Training 

$2.0 

$2.0 

$3.0 

- 

Summed  quantified  benefits 

$3.5 

$4.5 

$3.0 

$0.0 

Cell  entries  $  benefits  [($)  if  negative] 
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impact,  such  as  80  percent  (probability  is 
0.80  that  the  cost  impact  will  be  that 
amount  or  less)  can  also  be  assessed  as  a 
more  conservative  or  risk-averse  estimate. 
Both  the  most  likely  and  80  percent  cu¬ 
mulative  probable  estimates  can  be  used 
in  later  summaries.  It  should  be  indicated 
which  is  being  used,  and  use  should  be 
consistent. 

Step  6.  A  net  value  summary  table  is 
prepared.  The  “net  value”  is  a  number 
used  for  comparative  purposes  only  and 
will  not  correspond  to  a  true  price  or  life 
cycle  cost  (see  Table  4).  It  provides  for 
systematic  benefit  and  cost  comparisons 
that  take  account  of  not  only  the  realistic 
predicted  cost  to  the  government  of  the 
system  but  also  of  the  benefits  associated 
with  the  offerors’  respective  proposed  sys¬ 
tems.  Positive  quantified  benefits  are  sub¬ 
tracted  from  an  offeror’s  projected  cost  in 
this  table.  Risks  that  are  potential  expected 
losses  are  added  to  cost. 

That  offeror  with  the  lowest  net  value 
represents  the  Best  Value  at  this  point  in 
the  analysis.  The  quantified  benefits  have 
been  considered  in  the  analysis  and  the  net 
value  is  the  result.  The  nonquantified  ben¬ 


efits  must  still  be  considered,  and  these 
may  or  may  not  change  that  offeror  con¬ 
sidered  the  Best  Value. 

Step  7.  A  nonquantified  benefits 
evaluation  and  summary  is  prepared. 

This  is  done  for  those  benefits  and  risks 
listed  as  nonquantifiable  in  the  discrimi¬ 
nator  impacts  table.  The  nonquantifiable, 
or  qualitative,  discriminator  impacts  can 
then  be  detailed  in  another  table  in  the 
same  order  for  each  offeror.  See  Table  5 
for  an  example.  The  rank  order  can  be  a 
consensus  rank  order  developed  by  the 
SSAC  of  the  relative  importances  of  the 
nonquantified  discriminators  based  on  the 
descriptions  of  likely  impacts.  Since  quan¬ 
tification  of  impact  benefits  has  not  been 
accomplished,  precise  weights  cannot  be 
assessed.  However,  for  each  discrimina¬ 
tor,  the  SSAC  can  establish  a  rank  order 
of  offeror  proposals  in  terms  of  likely  im¬ 
pacts  with  respect  to  that  discriminator. 
These  rank  orders  are  indicated  in  the 
table. 

Comparison  to  quantified  benefits.  One 
method  for  further  precision  in  assessing 
the  relative  benefits  of  the  nonquantified 
benefits  is  a  comparison  of  each  of  these 


Table  4. 

Net  Value  Summary  Table  ($M) 


Offeror  X1 

Offeror  X2 

Offeror  X3 

Offeror  X4 

Original  bid  price 

$54.5 

$59.3 

$65.0 

$50.2 

Cost  realism 
adjustment  to  price 

+7.7 

+3.4 

+2.5 

+  10.8 

Most  probable  cost 
to  the  government 

62.2 

62.7 

67.5 

61.0 

Sum  of  quantified 
benefits  and  risks 

-3.5 

-4.5 

-3.0 

-0.0 

“Net  value”  of  offer 

$58.7 

;  :  . . . 

$64.5 

$61.0 
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Table  5. 

Nonquantified  Benefits  Table 


Discriminator 

Discriminator 

Offeror  XI 

Offeror  X2 

Offeror  X3 

Offeror  X4 

Impact  Area 

Benefit 

Benefit 

Benefit 

Benefit 

Explanation 

Explanation 

Explanation 

Explanation 

Key 

Productivity 

Strong 

Strong  key 

personnel 

mission 

selection  for 

personnel 

(KP) 

effectiveness 

program 

selections  for 

cost 

manager  and 

all  positions, 

software  j 

particularly 

development  ' 

software 

manager 

development 
manager  and 

Denote  as 

planning 

B(KP).  XI 

manager. 

Denote  as 
B(KP),  X2 

Past 

Productivity 

Extensive 

Relevant 

Relevant 

Relevant 

performance 

mission 

relevant 

experience 

experience 

experience 

(PP) 

effectiveness 

experience 

with  like 

with  like 

with  like 

with  like 

projects  and 

projects 

projects 

projects:  this 

this  agency 

and  this 

and  this 

is  offset  by 

agency,  but 

agency 

significant 

Denote  as 

history  of 

cost  or 

B(PP),  X2 

significant 

Denote  as 

schedule 

cost  or 

B{PP),X4 

overruns 

schedule 

overruns 

Denote  as 
B{PP).  XI 

Denote  as 
B(PP),  X3 

Position  of 
offeror  rank 
ordered 
benefits 

Rank  3 

Rank  1 

Rank  4 

Rank  2 

nonquantified  benefits  to  those  that  have 
been  quantified,  thus  bracketing  non¬ 
quantified  benefit  values  between  the  val¬ 
ues  of  those  that  are  quantified.  Which  is 
worth  more  to  the  government,  the 
nonquantified  benefit  or  the  quantified 
benefit?  For  a  particular  offeror  benefit,  a 
pair  of  quantified  benefits  is  found  such 
that  the  nonquantified  benefit  is  worth 
more  than  one  benefit  and  less  than  the 
other.  These  benefits  can  be  any  from  the 


li.st  of  quantified  benefits  (if  there  is  a  suf¬ 
ficiently  large  set).  Assign  the  nonquan¬ 
tified  benefit  a  dollar  value  between  the 
quantified  values.  If  desired,  these  could 
be  integrated  with  the  quantified  benefits 
in  the  net  value  table,  or  they  can  be  in¬ 
serted  in  the  nonquantified  benefits  table 
(Table  5). 

Direct  comparison  of  nonquantified 
benefits.  Nonquantified  benefits  can  also 
be  directly  compared  to  each  other  using 
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an  iterative,  stepwise  process.  There  are 
numerous  procedures  for  such  compara¬ 
tive  evaluations  involving  the  ordinal 
properties  of  the  benefits.  For  example, 
in  comparing  the  benefits  of  Offerors  1 
and  2,  first  the  relative  magnitudes  of  the 
nonquantified  benefit  items  of  each  off¬ 
eror  are  rank-ordered  relative  to  each  other 
without  comparing  to  those  of  the  other 
offeror.  Then  the  smallest  benefit  item  (or 
largest)  of  one  offeror  is  compared  to  the 
smallest  (or  largest)  benefit  item  of  the 
other.  A  decision  is  made  as  to  which  item 
is  worth  more  to  the  government  and  why. 
The  reasons  for  the  decision  are  recorded. 
Then,  for  the  offeror  whose  smallest  ben¬ 
efit  item  is  the  lesser  of  the  two,  his  small¬ 
est  two  benefit  items  are  compared  to  the 
smallest  of  the  other  offeror.  These  two 
must  either  be  worth  more  or  less  than  the 
smallest  item  of  the  other  offeror.  If  less, 
the  third  smallest  item  is  added.  If  larger, 
the  other  offeror’s  second  smallest  ben¬ 
efit  item  is  added  to  his  first  and  the  pro¬ 
cess  is  continued.  Eventually,  the  benefits 
of  each  offeror  will  be  ordered  relative  to 
those  of  the  other  offeror  and  there  will 
be  an  advantage  for  one  of  the  two.  This 
is  done  for  each  pair  of  offerors  (if  neces¬ 
sary).  Actually  there  will  be  many  short¬ 
cuts  in  this  seemingly  tedious  procedure, 
and  it  will  not  require  as  much  time  as  ini¬ 
tially  anticipated. 

A  comparison  of  the  nonquantified  ben¬ 
efits  listed  Table  5  indicates  that: 

B(PP),  X2  equals  approximately  B(PP), 
X4 

B(PP),  X4  is  significantly  greater  than 
B(PP),  XI 


B(PP),  XI  is  slightly  greater  than 
B(PP),  X3 

B(KP),  X2  is  greater  than  B(KP),  XI. 

These  inequalities  imply  that: 

B(PP),  X2  +  B(KP),  X2  >  B(PP),  XI  + 
B(KP),X1. 

Similarly,  further  direct  comparisons  of 
these  nonquantified  benefits  indicate  that 

B(PP),  X4  >  B(PP),  XI  -I-  B(KP),  XI 

B(PP),  XI  -I-  B(KP),  XI  >  B(PP),  X3. 

By  aggregating  the  inequalities,  it  can 
be  deduced  that  the  order  of  the 
nonquantified  benefits  of  the  offerors  is 
X2  -  X4  -  XI  -  X3  as  indicated  in  the 
bottom  row  of  Table  5.  Because  Bidder 
X2  has  the  best  overall  net  value  prior  to 
the  nonquantified  benefits  comparison, 
and  given  the  dominance  by  X2  in  the 
nonquantified  benefits,  X2  remains  the 
best  overall  net  value. 

Step  8.  Quantified  and  nonquantified 
benefits  are  traded  off.  After  the 
nonquantified  benefits  are  compared,  one 
of  two  situations  will  exist.  The  offeror 
with  the  lowest  quantified  net  value  may 
also  have  the  greatest  nonquantified 
benefits  ranking.  If  so,  the  process  is 
finished  and  that  offeror  is  recom¬ 
mended.  If  the  offeror  with  the  greatest 
total  nonquantified  benefits  ranking 
does  not  have  the  lowest  net  value, 
nonquantified  benefits  must  be  traded 
against  quantified  benefits  using  the 
process  described  in  Step  7. 

The  comparison  can  be  accomplished 
in  several  ways.  One  way  is  to  again  pro- 
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ceed  pairwise.  The  offerors  are  ordered  in 
terms  of  the  net  value  results.  The  process 
starts  with  the  lowest  net  value  offeror.  The 
net  value  difference  between  this  and  the 
second  offeror  is  a  benefit  advantage  over 
the  second  offeror.  This  net  value  advan¬ 
tage,  in  dollars,  is  compared  to  the 
nonquantified  benefits  directly.  The  net 
value  difference  may  be  more  beneficial 
than  the  total  of  all  of  the  nonquantified 
benefits  of  the  second  offeror  or  it  may  be 
possible  to  bracket  it  somewhere  in  the 
order.  This  pairwise  comparison  is  accom¬ 
plished  iteratively  and  the  results  are  in¬ 
tegrated,  interpreted,  and  summarized. 
The  nature  of  the  procedure  should  be 
fairly  obvious:  we  won’t  discuss  further 
variations.  The  point  is  that  this  compara¬ 
tive  procedure  provides  a  linked  evalua¬ 
tion  with  the  paired  comparison  ordering 
and  reasons  as  supporting  rationale  for  the 
results. 

We  can  draw  two  conclusions.  One  is 
that  this  benefit  comparison  procedure  is 
systematic,  and  although  it  depends  only 
on  ordinal  preferences,  it  can  be  strongly 
supported  by  the  benefits  rationale  pre¬ 
pared  in  the  benefits  summary  table.  The 


second  is  that  the  procedure  is  guaranteed 
to  produce  a  recommendation.  Though  the 
procedure  may  seem  very  tedious,  it  can 
be  made  quite  efficient.  The  important 
point  is  that  ordinal  ranking  procedures 
that  do  not  assume  inordinate  levels  of 
precision  can  be  used  to  produce  the  sys¬ 
tematic  evaluation  desirable  in  Best  Value 
procurements.  The  criticism  that  such  as¬ 
sessments  cannot  be  defended,  and  there¬ 
fore  should  not  be  employed,  does  not 
make  sen.se.  If  such  judgments  cannot  be 
made,  how  can  the  decision  made  using 
some  other  less  systematic  approach  be 
defended? 

Step  9.  An  overall  best  value  sum¬ 
mary  table  is  prepared.  Finally,  depend¬ 
ing  on  the  actual  process  u,sed  to  compare 
nonquantified  benefits,  the  BVWG  must 
judge  whether  the  results  of  this  benefits 
comparison  should  change  the  offeror  or¬ 
der  in  the  net  value  table  (see  Table  6).  A 
summary  final  rank  order  table  is  prepared 
indicating  the  nature  of  the  final  order  with 
the  systematic  explanation  of  the  benefits 
and  costs  audit  trials  that  re.sulted  in  the 
ordering.  This  table  contains  the  original 


Table  6. 


Best  Value  Summary  Table  ($M) 


Offeror  XI 

Offeror  X2 

Offeror  X3 

Offeror  X4 

Original  bid  price 

S54.5 

$59.3 

$65.0 

$50.2 

‘Cost  realism'  cost 

$62.2 

$62.7 

$64.5 

$61.0 

Net  value  adjusted  for 
quantified  benefits 

$58.7 

$58.2 

$64.5 

$61.0 

Nonquantified  benefits 
comparison  results 

Third  highest 
aggregated 
nonquantified 
benefits 

Highest 

aggregated 

nonquantified 

benefits 

Fourth  Highest 
(lov^rest) 
aggregated 
nonquantified 
benefits 

Second  Highest 
aggregated 
nonquantified 
benefits 

Summary 

Rank  2  or  3 

Best  Net  Value 

Least  Value 

Rank  2  or  3 
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proposed  price,  the  adjusted  “cost  realism” 
cost,  the  net  value  including  quantified 
discriminator  impact,  and  the  nonquan- 
tified  benefits  from  the  nonquantified  ben¬ 
efits  table.  The  iterative  stepwise  process 
described  above  to  reach  the  final  deci¬ 
sion  can  be  explained  proceeding  through 
the  related  comparisons  using  this  sum¬ 
mary,  with  final  remarks  in  the  comments 
area.  For  the  example  here,  the  results  of 
the  comparative  process  are  clear  with 
respect  to  the  best  and  worst  overall  net 
values.  Because  X2  is  the  best  overall  net 
value,  it  is  not  necessary  to  resolve  the 
uncertainty  about  the  ranks  of  X 1 ,  and  X4. 
(The  quantified  overall  cost  (net  value)  for 
X4  is  $2.3M  higher  than  that  for  XI,  but 
X4  nonquantified  benefits  are  higher  than 
those  for  XI.  The  question  concerns  the 
quantified  value  of  these  benefits.  Do  they 
compensate  for  the  $2.3M?  If  this  ques¬ 
tion  needed  resolution,  the  process  de¬ 
scribed  earlier  would  be  used.) 


Legal  Precedents;  Protests  of 
Best  Value  Decisions 


Protests  involving  Best  Value  source  se¬ 
lections  have  involved  issues  of  impropri¬ 
eties,  unfairness,  improper  procedures, 
failure  to  follow  advertised  procedures, 
and  others.  Several  protests  of  Best  Value 
procurements  have  involved  the  conten¬ 
tion  that  the  government  didn’t  tell  the 
offerors  what  it  was  going  to  do  in  suffi¬ 
cient  detail  to  allow  them  to  clearly  state 
their  case  in  the  most  advantageous  way, 
or  that  the  government  did  not  do  what  it 
said  it  would  do.  The  following  are  dis¬ 
cussions  of  several  relevant  protests  in¬ 
cluding  non-DoD  examples.  (See  GAO 


Review,  April,  1994;  GSBCANo.  12813- 
P-R  {LEXIS  255},  1994;  Koch,  1994; 
O’Keefe,  1978;  and  Sochon,  1994,  for 
examples  and  discussion,  and  Widnall  v. 
B3H  Corp,  1996).)  These  protests  dem¬ 
onstrate  that  the  problem  of  an  appropri¬ 
ate  procedure  for  the  crucial  benefit  to  cost 
tradeoffs,  the  resolution  of  which  should 
include  practicing  decision  analysts,  is  in 
danger  of  being  determined  by  legal  pre¬ 
cedents  with  respect  to  protests. 

Important  Best  Value  Cases 

Cost  realism  adjustments.  The  Trea¬ 
sury  Multi-User  Acquisition  Contract 
(TMAC)  (Koch,  D.,  1994)  decisions  es¬ 
tablished  several  precedents  for  Best  Value 
procurements.  One  was  that  adjustments 
for  cost  realism  and  the  projection  of  most 
probable  life  cycle  cost  to  the  government 
were  acceptable  practices.  Also,  there  is 
no  requirement  in  the  law  for  technical 
factors  to  be  proportional  to  cost  factors. 
Technical  factors  can  have  a  cost  of  their 
own  expressed  in  terms  of  dollars  irrespec¬ 
tive  of  cost  of  acquisition  and  ownership 
considerations.  TMAC  also  established 
that  if  no  offeror  protests  the  lack  of  in¬ 
formation  in  the  solicitation  on  the  evalu¬ 
ation  procedures  prior  to  proposal  submis¬ 
sion,  then  there  is  no  basis  for  a  later  pro¬ 
test  based  on  this  lack  of  information. 

Quantification.  In  Grumman  Data 
Systems  Corporation  v.  Department  of  the 
Air  Force  ( 1 992,  Gmmman),  the  Air  Force 
used  quantifiable  and  nonquantifiable  dis¬ 
criminators.  (See  Federal  Contract  Report, 
1994.)  The  quantifiable  discriminators 
were  translated  to  dollars.  The  importance 
of  the  nonquantifiable  discriminators  was 
stressed  but  not  translated  to  dollars.  This 
ruling  upheld  this  version  of  tradeoffs  but 
does  not  eliminate  others.  This  tradeoff 
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process  was  also  upheld  in  Computer  Sci¬ 
ence  Corporation  (CSC)  v.  Department  of 
the  Army  in  the  Army  Reserve  Compo¬ 
nent  Automation  Systems  (RCAS)  pro¬ 
curement  (see  Koch,  D.  1994,  and  GSBCA 
No.  1I635-P.  1992  B.P.D.  §  100).  This 
Grumman  protest  also  establi.shed  the  rea¬ 
sonableness  of  the  cost  technical  tradeoff 
(CTTO)  analysis  by  stating,  “There  is  no 
formulaic  methodology  for  conducting  a 
Best  Value  determination;  what  matters  is 
that  the  award  is  consistent  with  the  terms 
of  the  solicitation  and  that  any  price  pre¬ 
mium  is  justified 
by  the  specific 
technical  en¬ 
hancements.” 
(See  Federal 
Contract  Report, 
1994.)  Thus 
there  is  no  nec¬ 
essary,  defined 
fonnula  for  link¬ 
ing  technical 
scores  and  co.st. 
This  link  must 
be  assessed.  Be¬ 
cause  there  is 
not  one  prescribed  formula,  other  criteria 
such  as  “reasonable  approach”  and  “even- 
handedness”  are  appropriate  for  evaluat¬ 
ing  the  linkage  approach. 

In  Lockheed  Missiles  and  Space  Com¬ 
pany  V.  Department  of  the  Treasury, 
TMAC  II  (GSBCA  Nos.  1 1776-P,  11777- 
P,  1992,  B.P.D.  §  155),  the  government 
failed  to  properly  conduct  present  value 
and  most  probable  co.st  adjustments  on  the 
estimated  dollar  value  of  the  increased 
technical  benefits  in  the  Best  Value  assess¬ 
ment.  but  had  done  so  on  the  original  cost 
comparison.  Lockheed  recomputed  the 


adjusted  most  probable  costs  based  on  the 
formulas  u.sed  in  the  source  selection,  and 
the  results  favored  a  reversal  of  the  deci¬ 
sion.  Nonetheless,  the  protest  was  denied. 
The  General  Services  Board  of  Contract 
Appeals  (GSBCA)  determined  that  there 
were  other  values  (e.g.,  nonquantified  dis¬ 
criminator  benefits)  that  the  Treasury  De¬ 
partment  did  not  include  in  their  analysis, 
and  that  these  compensated  for  the  error 
in  benefits  assessment.  Note  that  this  es¬ 
sentially  means  that  GSBCA  was  itself  as¬ 
sessing  the  value  of  benefits  and  also  de¬ 
termining  that  nonquantified  benefits  do 
trade  off  against  quantified  benefits!  The 
TMAC  II  ruling  also  upheld  the  use  of  the 
price-risk  analysis  discussed  earlier. 

Clarity  with  respect  to  benefit 
tradeoff  procedures.  The  protest  by  Sys¬ 
tem  Re,sources  Inc.  (GSBCA  No.  12536- 
P,  1993,  B.P.D.  §  253)  was  upheld,  indi¬ 
cating  that  simply  stating  in  the  RFP  that 
a  Best  Value  approach  will  be  used  is  not 
enough  guidance.  It  must  also  be  indicated 
that  a  benefit-cost  tradeoff  will  occur.  In 
this  case,  credit  was  given  to  an  offeror’s 
proposal  for  projected  capability  in  excess 
of  a  threshold  standard  listed  in  the  RFP, 
and  offerors  were  not  told  that  any  such 
credit  would  be  given.  The  RFP  can  be 
silent  on  exactly  how  this  trade  benefit- 
cost  tradeoff  will  be  done,  but  this  silence 
may  increase  the  probability  of  a  protest. 
In  this  case,  the  lack  of  instructions  was 
misleading,  for  the  standard  was  stated  in 
threshold  form  with  no  indication  that  ca¬ 
pability  above  that  threshold  would  be 
traded  against  capabilities  in  other  areas 
(i.e.,  explanation  of  the  SSEB  conditional 
decision  rule  for  trading  off  capability 
above  this  threshold). 


"There  is  no  formu¬ 
laic  methedelegy  for 
cenducfing  a  Best 
Value  determination; 
what  matters  is  that 
the  cnvard  is  consis¬ 
tent  urith  the  terms 
of  the  solicitation 
and  that  any  price 
premium  is  justified 
by  the  specific 
technical  enhance¬ 
ments." 
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Benefits  traceable  to  evaluation  fac¬ 
tors.  In  the  USAF  Desktop  IV  case  in  May 
1993,  it  was  stated  that  the  evaluator  can¬ 
not  assign  benefit  to  services  or  capabili¬ 
ties  (including  using  them  as  discrimina¬ 
tors  in  Best  Value)  not  specified  some¬ 
where  in  the  RFP  (O’Keefe,  1993).  The 
services  must  be  requested  or  identified 
in  some  useful  language  in  the  RFP.  Note 
that  the  same  ruling  established  that  ser¬ 
vices  that  are  inherent  but  unstated  can  be 
used  and  assigned  a  value  (Federal  Acqui¬ 
sition  Report,  1993).  The  Desktop  IV  rul¬ 
ing  also  established  that  the  National  De¬ 
fense  Authorization  Act  of  1991  gives 
defense  agencies  the  authority  to  make 
Best  Value  awards  without  discussions 
with  offerors — again  no  change  from  es¬ 
tablished  practice  (O’Keefe,  1993). 

Burden  of  proof  for  compliance  with 
solicitation  terms.  In  a  recent  and  impor¬ 
tant  decision,  the  GSBCA  upheld  a  pro¬ 
test  by  the  B3H  Corporation  against  the 
U.S.  Air  Force  (GSBCANo.  12813-P-R, 
1994  GSBCA  LEXIS  255  )  on  the  basis 
that  the  Air  Force  made  the  award  con¬ 
trary  to  the  terms  of  the  solicitation,  indi¬ 
cating  that  the  record  does  not  “with  any 
degree  of  certainty”  support  the  conclu¬ 
sion  that  the  added  values  of  the  awardees’ 
proposals  were  worth  the  higher  prices 
associated  with  those  respective  propos¬ 
als.  The  protest  was  by  a  small  business 
that  protested  only  the  small  business 
award  of  a  multiple-award  procurement, 
contending  that  the  protester  had  the  low¬ 
est  cost  of  the  small  business  proposals, 
and  that  the  Air  Force  selection  process 
was  inconsistent  with  the  terms  of  the  so¬ 
licitation.  It  was  claimed  that  the  Air  Force 
did  not  show  that  the  proposals  of  the  two 
small  business  offerors  awarded  contracts 


were  worth  the  extra  15  percent  and  5  per¬ 
cent  costs  respectively  associated  with 
them.  Thus  the  B3H  Corp.  maintained  that 
its  proposal  represented  the  best  value  to 
the  government.  The  decision  of  the 
GSBCA  supported  this  contention,  indi¬ 
cating,  among  other  things,  the  following: 
“What  is  lacking  as  a  whole  is  a  reasoned 
basis  leading  to  the  conclusion  that  the 
benefits  of  the  awardees’  proposals  are  in 
fact  worth  the  apparent  extra  costs.” 

The  single  dissenting  judge  (one  of 
three)  to  the  GSBCA  decision  summarized 
his  view  of  the  implication  of  the  GSBCA 
decision  to  uphold  the  protest.  “There  is 
also  the  matter  that  the  majority  seems  to 
think  that  the  Air  Force,  when  challenged, 
is  required  to  prove  that  its  procurement 
is  perfect.  In  ev¬ 
ery  legal  system  «|||  every  legal 
of  which  I  am  system  of  which  I 
aware,  quasi  or  am  aware,  quasi  Or 
otherwise,  he  otherwise,  he  who 
who  alleges  an  alleges  an  ImprOpri- 
impropriety  «»y  ■"«**  produce 
must  produce  evidence  sufficient 
evidence  suffi-  *•  P*'e'ee  it. 
cient  to  prove  it. 

This  majority,  for  whatever  reason,  has 
reversed  that  fundamental  rule.  In  this  pro¬ 
curement,  the  protester  said  it  didn’t  think 
the  two  better  qualified  offerors  were 
worth  the  additional  money,  but  it  offered 
no  evidence  at  all  to  prove  it,  thus  com¬ 
mitting  the  same  sin  that  the  majority  says 
that  the  government  committed,  and  which 
the  majority  used  as  a  reason  to  grant  the 
protest.” 

In  Widnall  v.  B3H  Corp.  (1996),  the 
B3H  decision  was  overturned  by  the  U.S. 
Circuit  Court  of  Appeals,  which  stated  that 
if  the  GSBCA  board  task  is  to  assure  that 
an  agency’s  procurement  decision  is 
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grounded  in  reason,  and  if  such  is  the  case, 
then  the  board  defers  to  the  agency  deci¬ 
sion.  even  if  the  board  might  have  chosen 

a  different  off¬ 
eror  (i.e.,  there 
is  not  necessar¬ 
ily  a  need  to  be 
right,  only  to  be 
reasonable). 
The  court  fur¬ 
ther  noted  that 
discriminators 
can  be  quanti¬ 
fied  or  non- 
quantified  “for 
the  board  does  not  require  that  each  dif¬ 
ference  in  a  proposal  be  assigned  an  exact 
dollar  value  representing  its  worth  to  the 
government.”  (Widnall  v.  B3H,  1996,  part 
III).  The  agency  is  required  to  present  a 
reasoned  analysis  showing  that  the  gov¬ 
ernment  expects  to  receive  benefits  com¬ 
mensurate  with  the  extra  costs  it  will  have 
to  pay.  This  case  also  goes  on  to  describe 
other  decisions  entrusted  to  the  agency  and 
thus  the  SSA  including,  for  example, 
which  nonquantified  discriminator  to  em¬ 
phasize.  Thus,  this  decision  removes  the 
apparent  need  for  the  agency  to  prove  that 
its  analysis  is  “correct”  beyond  some  rea¬ 
sonable  doubt. 

Implications  of  Best  Value 
Protest  Findings 

There  are  .several  implications  of  the.se 
findings,  and  unfortunately  they  are  not 
unambiguous.  One  is  that  there  has  not 
been  agreement  on  what  constitutes  suffi¬ 
cient  demonstration  of  Be.st  Value.  It  is 
very  obvious  that  no  one  is  asking  for  a 
“perfect”  analysis,  as  was  clearly  demon¬ 
strated  in  the  TMAC II  decision.  However, 
as  demonstrated  clearly  by  B3H,  some 


systematic  demonstration  of  the  benefit  to 
cost  tradeoffs  beyond  a  mere  statement  of 
opinion  by  the  SSA  is  required.  Even  the 
dis.senting  judge  in  the  B3H  prote.st  would 
probably  not  argue  with  this.  The  ques¬ 
tion  .seems  to  regard  what  constitutes  suf¬ 
ficient  rigor  with  re.spect  to  e.stablishing 
the  relative  worth  of  any  benefits  claimed 
for  an  offeror  who  is  not  the  lowest  bid¬ 
der.  Systematic  dollar  value  quantification 
of  impact  area  benefits  using  rigorous 
quantification  procedures  and  mea.sures 
ba.sed  on  historical  precedent  will  likely 
provide  sufficient  proof  However,  it  is 
generally  not  clear  what  parts  of  benefit 
can  be  translated  to  dollars.  The  B3H  de¬ 
cision  demonstrated  that  the  SSA’s  de¬ 
tailed  claims  regarding  increased  savings 
to  accrue  may  not  be  sufficient  (at  least  to 
some  judges).  But  Widnall  v.  B3H  dem¬ 
onstrates  that  a  “reasoned  analysis”  is  suf¬ 
ficient.  Would  a  .sufficient  ju.stification  be 
a  systematic,  linked  tradeoff  process  such 
as  that  used  in  the  Multi-Attribute  Utility 
Theory  (MAUT)  technique  with  a  judg¬ 
mental  assessment  of  the  benefit-to-cost 
link  supported  by  logic  and  rationale?  This 
is  more  sy.stematic  than  extensive  testi¬ 
mony  by  the  SSA  in  B3H,  but  it  is  less 
rigorous  than  detailed  cost  modeling  of  the 
increased  benefits  using  precedent-based 
measures.  This  question  is  not  clearly  an¬ 
swered  by  these  ca.ses,  but  this  is  certainly 
a  reasoned  analysis  that  satisfies  the  cri¬ 
teria  of  Widnall  v.  B3H. 

Another  finding  that  is  apparent  from 
these  cases  is  that  the  government  should 
be  very  clear  in  its  guidance  to  offerors, 
specifying  in  advance  the  procedures  it 
intends  to  follow  and  even  its  conditional 
decision  rules  where  feasible.  The  govern¬ 
ment  should  specify  not  only  that  a  Best 
Value  procurement  will  take  place,  but  that 


"The  question  seems 
to  regard  what 
constitutes  sufficient 
rigor  with  respect  to 
establishing  the 
relative  worth  of 
any  benefits  claimed 
for  an  offeror  who 
is  not  the  lowest 
bidder" 
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cost  and  technical  factors  will  be  traded 
off.  An  example  in  support  of  this  prescript 
is  the  U.S.  Marine  Corps  Mobile  Protected 
Weapon  System  (MPWS)  procurement.  In 
that  concept  design  procurement  the  gov¬ 
ernment  provided  the  offerors  the  actual 
evaluation  structure,  in  that  case  a  MAUT 
model,  including  the  attributes  (called  fac¬ 
tors  or  criteria)  and  intra-attribute  utility 
functions,  attribute  weights  and  rationale 
for  all.  The  evaluation  was  quite  success¬ 
ful  (see  Buede  &  Bresnick,  1992). 

The  government  should  obviously  fol¬ 
low  the  procedures  specified  in  the  SSEP 
and  explained  to  offerors  in  the  solicita¬ 
tion  and  should  avoid  any  procedures  not 
so  specified.  But  the  government  cannot 
specify  all  possible  contingencies.  Further, 
it  cannot  specify  the  exact  details  that  will 
determine  a  decision  outcome.  It  can 
clearly  define  the  evaluation  factors  and 
subfactors  (if  any)  and  specify  how  the 
evaluations  will  be  conducted  with  respect 
to  these.  The  specific  Best  Value  method¬ 
ological  approach  to  be  used  can  be  clearly 
laid  out  in  the  solicitation.  In  current  prac¬ 
tice,  however,  there  is  not  apparent  agree¬ 
ment  on  a  single  recommended  approach. 
An  urgent  job  for  decision  analysts  is,  at  a 
minimum,  to  lay  out  guidelines  for  sev¬ 
eral  conditional  approaches  along  with  the 
conditions  under  which  each  approach 
should  be  employed.  Unless  this  is  done, 
the  solution  may  well  be  determined  by 
legal  experts  having  only  a  partial  under¬ 
standing  of  the  measurement  concepts  and 
resultant  implications  of  their  decisions. 

The  thing  that  is  unclear  from  the  legal 
discussion  is  the  specific  nature  of  the 
benefit-to-cost  tradeoff.  The  B3H  case 
demonstrates  that  a  mere  verbal  descrip¬ 
tion  of  benefits  by  the  SSA  is  probably 
not  sufficient  justification  for  decisions. 


Other  protest  outcomes  (e.g.,  Widnall  v. 
B3H)  seem  to  demonstrate  that  an  attempt 
at  a  systematic  approach  to  characterize 
benefits  and  offeror  differences  with  re¬ 
spect  to  discriminator  impacts  should  suf¬ 
fice.  The  use  of  MAUT  tools  to  aid  the 
SSAC  in  this  systematic  tradeoff  process 
should  be  valuable.  Such  a  MAUT  ap¬ 
proach  would  be  hardest  to  defend  if  it 
relied  only  on  judgmental  assessments. 
These  judgmental  assessments  should  be 
systematically  linked  by  the  MAUT  pro¬ 
cess  and  supported  by  rationale,  and  thus 
they  would  be  better  than  SSA  opinions 
about  the  group  of  discriminators  (the  ap¬ 
parent  problem  in  B3H). 

If  the  benefit  tradeoffs  were  further  sup¬ 
ported  by  a  linkage  to  cost  realism  esti¬ 
mates  through  a  systematic  analysis  or  al¬ 
gorithm,  the  MAUT  analysis  would  be 
very  strong  and 

"An  urgent  job  for 
decision  analysis  is, 
at  a  minimum,  to  lay 
out  guidelines  for 
several  conditional 
approaches  along 
«dlh  the  conditions 
under  which  each 
approach  should  be 
employed ;  .f;::; 


should  provide 
the  “reasoned 
analysis”  re¬ 
quired  by  the 
GSBCA.  The 
MAUT  analysis 
will  generally 
not  take  as 
much  time  as  a 
detailed  costing 


of  discriminator 

impact  areas  using  rigorous  costing  algo¬ 
rithms.  The  SSAC  and  BVWG  generally 
do  not  plan  to  spend  several  weeks  or 
months  doing  the  Best  Value  analysis.  This 
would  imply  that  costing,  if  done,  would 
involve  simplified  approximations  to  more 
formal  approaches.  Otherwise,  MAUT 
provides  a  reasoned  analysis  that  can  be 
accomplished  in  a  shorter  time. 

One  answer  to  the  desirable  nature  of 
the  benefit-to-cost  tradeoff  process  is,  “It 
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depends.’’  That  is.  sometimes  tradeoffs  can 
be  quantified;  sometimes  they  cannot. 
Measurement  always  involves  abstraction 
of  the  real  world  situation  to  a  mathemati¬ 
cal  model,  and  the  models  appropriate  for 
different  conditions  are  different.  The  lack 
of  a  universal  answer  may  be  dissatisfy¬ 
ing  to  some,  but  it  should  comfort  the  prac¬ 
titioners  who  have  always  shied  away 
from  quantification  because  they  feared 
rigid  application  of  inappropriate  models. 
It  should  also  satisfy  the  practitioners  who 
wish  to  plan  ahead,  for  it  implies  that  the 
entire  acquisition  must  be  systematically 
implemented,  with  linked  modeling  and 
measurement  procedures  defined  early 
and  consistently  implemented.  This  can 
help  prepare  for  later  quantification  of 
benefits. 

A  general  summary  of  this  guidance  is 
that  when  the  source  selection  is  done,  the 
source  selection  team  must  be  able  to  con¬ 
vincingly  show  that  the  analysis  and  de¬ 
cision-making  process  were  sound  and 
fair.  An  SSA  opinion  that  “the  added  co.st 
is  worth  it”  will  probably  not  .suffice  (and 
did  not  initially  in  the  referenced  B3H 
case).  All  costs  and  benefits  must  be  sys¬ 
tematically  characterized  as  quantified  or 
not.  and  the  decision  process  must  have  a 
systematic,  undenstandable,  and  consi.stent 
logic  thread.  Unquantified  benefits  can  be 
traded  against  cost  differences,  but  the 
trades  must  be  rigorous,  understandable, 
and  the  result  of  a  “reasoned  analysis,’’  a 
logically  consistent  decision  process. 
While  there  may  be  some  residual  uncer¬ 
tainty  about  the  absolute  precision  of  as¬ 
sessments  used  in  evaluations,  the  proce¬ 
dures  were  applied  to  all  offerors  in  an 
evenhanded  manner. 

Decision  support  tools  help  to  provide 
the  necessary  consistency  and  audit  trail. 


The  use  of  these  tools  provides  for  public 
scnitiny  of  the  evaluation  process  and 
avoids  an  inherent  weakness  in  the  argu¬ 
ment  of  the  dissenting  judge  in  the  B3H 
protest  case.  That  judge  seemed  to  claim 
that  the  SSAC  shouldn’t  be  second 
guessed  because  the  board  is  most  knowl¬ 
edgeable  on  all  the  issues  and  thus  best 
qualified  to  make  the  tradeoff  assessments. 

However,  while  the  SSAC  members 
may  be  quite  knowledgeable,  this  doesn’t 
guarantee  the  sanctity  or  even  the  correct- 
ne.ss  of  their  procedures.  An  “analysis 
based  in  reason  is  still  required.”  (See 
O’Connor,  Paris,  and  Lovelace  [  1 996]  for 
further  di.scussion  of  such  analyses  and 
tools  to  support  them.) 

Recommendations  Based  on  Best 
Value  Applications 


The  authors  have  collectively  applied 
the  principles,  procedures,  and  tools  dis- 
cus.sed  in  this  article  to  more  than  100  ac¬ 
quisitions,  many  of  which  have  involved 
Best  Value  source  selections.  This  section 
provides  recommendations  resulting  from 
that  Best  Value  work. 

Time: 

Underestimation  and 
Resultant  Rush  to  Judgment 

All  of  the  historic  problems  of  planning 
time  pertain  to  acquisitions.  More  often 
than  not,  the  evaluation  team  develops  an 
evaluation  schedule  (and  program  sched¬ 
ule)  ba.sed  on  overly  optimistic  estimates 
of  the  time  needed  to  accomplish  the  ac¬ 
tivities  involved  in  executing  the  source 
selection  plan.  The  reasons  are  many  and 
are  well  known  to  practitioners.  For  ex¬ 
ample,  the  time  required  for  the  Source 
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Selection  Evaluation  Board  to  accomplish 
its  evaluation  is  very  often  longer  than 
planned.  The  time  required  for  discussions 
with  offerors  including  clarification  re¬ 
quests,  deficiency  ratings,  oral  discus¬ 
sions,  and  revision  of  offers  is  also  very 
often  longer  than  anticipated.  Yet,  because 
acquisitions  are  usually  highly  visible  to 
the  government  and  public,  and  delays  in 
awards  can  be  costly  to  all,  the  planned 
award  announcement  date  is  maintained 
until  it  is  obviously  impossible  to  attain. 
Usually,  rather  than  slip  that  date,  the  time 
allowed  for  activities  at  the  end  of  the 
cycle  are  squeezed.  In  Best  Value  source 
selections,  this  squeezed  time  is  the  time 
for  the  Best  Value  working  group  to  ac¬ 
complish  Best  Value  deliberations  and  dis¬ 
criminator  quantification.  This  problem  is 
often  further  exacerbated  by  insufficient 
time  allotment  to  these  activities  (espe¬ 
cially  discriminator  quantification)  in  the 
original  schedule.  This  insufficiency  in  the 
original  scheduling  can  be  caused  by  a 
lack  of  understanding  of  both  what  is  in¬ 
volved  in  Best  Value  source  selections  and 
of  the  time  required  to  implement  a  valid 
cost-benefit  tradeoff  process  and  associ¬ 
ated  discriminator  impact  benefit  quanti¬ 
fications. 

The  validity  and  precision  of  the  cost- 
benefit  tradeoff  process  and  associated 
discriminator  impact  benefit  quantifica¬ 
tion  are  directly  affected  by  the  time  al¬ 
lotted  to  them.  Procedures  to  assure  va¬ 
lidity,  precision,  and  thus  “correctness” 
take  time.  The  recommendation  here  is 
obvious.  Sufficient  time  should  be  allot¬ 
ted  to  the  Best  Value  deliberations  in  the 
original  source  selection  schedule,  and  that 
time  should  not  be  shortened  unless  it  be¬ 
comes  clear  that  it  is  not  required,  (e.g.,  a 
Best  Value  situation  does  not  eventuate 


after  SSEB  evaluation).  As  a  general  rule, 
at  least  one  month  should  be  allotted  to 
discriminator  impact  benefit  quantification. 

The  authors  have  often  faced  the  argu¬ 
ment  that  there  is  no  need  for  benefit  quan¬ 
tification,  especially  if  it  is  going  to  be 
done  poorly,  and  that  traditional  source  se¬ 
lection  procedures  will  suffice.  The  ben¬ 
efits  of  the  Best  Value  approach  will  not 
be  argued  here,  save  to  note  that  it  was 
originally  de¬ 
veloped  to  ad¬ 
dress  an  appar¬ 
ent  inadequacy 
in  the  tradi¬ 
tional  process. 

The  issue  that 
the  traditional 
process  should 
be  implemented 
because  practi¬ 
tioners  do  not 
plan  and  imple¬ 
ment  well  is  not  really  a  viable  one.  If  the 
intent  to  use  a  Best  Value  process  has  been 
stated  to  the  offerors,  then  it  should  be 
implemented  as  well  as  can  be  done. 


"Sufficient  lime 
•houid  be  aliened  to 
the  Best  Value  delib¬ 
erations  in  the 
original  source 
selection  schedule, 
and  that  lime  should 
not  be  shortened 
unless  it  becomes 
clear  that  it  is  not 
required." 


Clarity  of  Intent  to  Use 
A  Best  Value  Process 

Another  important  issue  is  that  it  should 
be  clear  both  to  offerors  and  the  acquisi¬ 
tion  team  that  a  Best  Value  process  will 
be  used.  Vague  wording  such  as  “the  gov¬ 
ernment  will  choose  the  offeror  provid¬ 
ing  the  best  overall  value  to  the  govern¬ 
ment”  are  considered  by  some  practitio¬ 
ners  to  adequately  signal  a  Best  Value  ac¬ 
quisition  to  the  offerors.  For  the  offerors, 
a  less  ambiguous  statement  is  one  that  in¬ 
dicates  that  the  government  will  imple¬ 
ment  a  Best  Value  process  that  will  involve 
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a  cost-benefit  tradeoff  process.  For  the 
government,  the  SSAC  members  should 
understand  the  Best  Value  process  that  will 
be  used  and  should  plan  accordingly  for  it. 

Best  Value  Working  Group 
Composition  and  Procedure 

A  BVWG  chairperson  should  be  ap¬ 
pointed  as  early  in  the  evaluation  process 
as  feasible.  This  provides  time  for  that 
person  to  organize  for  such  issues  as 
BVWG  composition,  schedule,  and  pro¬ 
cedures  as  well  as  to  begin  to  attack  tech¬ 
nical  issues  such  as  procedures  and  tools 
for  cost-benefit  tradeoffs  and  discrimina¬ 
tor  impact  benefit  quantification.  The 
BVWG  should  include  a  sufficient  num¬ 
ber  of  SSAC  personnel  to  assure  valid 

benefit  assess¬ 
ments  and  also 
SSAC  buy-in 
into  the  analysis 
results.  At  the 
same  time,  the 
BVWG  should 
not  be  so  large 
that  meetings 
are  too  large  and  progress  difficult.  All 
BVWG  members  should  expect  to  be 
workers  that  will  participate  totally  in  the 
Best  Value  cost-benefit  tradeoff  process. 
The  larger  this  group,  the  more  important 
it  is  that  the  meeting  leader  (BVWG  chair¬ 
person  or  a  facilitator)  possess  both  good 
meeting  facilitation  and  analysis  skills. 

The  BVWG  should  also  include  evalu¬ 
ation  personnel  most  familiar  with  the 
content  and  evaluation  of  the  offerors’  pro¬ 
posals.  This  familiarity  includes  knowl¬ 
edge  of  the  basis  for  assessed  costs  and 
assessed  technical  and  management  ben¬ 
efits  associated  with  the  respective  pro¬ 
posals.  Such  personnel  include  Source 


Selection  Evaluation  Board  (SSEB)  mem¬ 
bers  such  as  the  technical,  management, 
pa.st  performance,  and  cost  panel  chairper¬ 
sons.  These  personnel  can  serve  as  BVWG 
advisors. 

Discriminators: 

How  Many? 

The  number  of  di.scriminators  needed 
is  often  pondered .  The  answer  depends  on 
the  particular  source  selection  and  will 
evolve  from  the  proposal  evaluation  pro- 
ce,ss  and  results.  A  set  of  discriminators 
must  be  established  that  validly  charac¬ 
terizes  the  benefit  advantages  of  each  of 
the  proposals  over  others.  The  discrimi¬ 
nators  must  not  necessarily  span  all  of  the 
benefits,  but  rather  must  characterize  the 
benefits  that  discriminate  in  a  discernible 
way  among  proposals.  Thus  benefits  com¬ 
mon  to  all  need  not  be  reexamined  at  this 
point. 

All  di.scriminators  must  be  traceable 
back  to  one  or  more  evaluation  factors. 
This  doesn’t  mean  that  they  correspond 
to  a  specific  set  of  evaluation  score  dif¬ 
ferences,  but  they  must  be  inherent  in  the 
intent  of  the  evaluation  or  must  be  deriv¬ 
able  as  implications  of  the  evaluation.  (No 
new  evaluation  factors  representing  ser¬ 
vices  or  capabilities  not  requested  in  the 
solicitation  should  be  introduced  at  this 
point.  This  is  an  important  and  often  dif¬ 
ficult  issue  that  can  be  misunderstood.) 
Di,scriminator  impact  areas  must  be  iden¬ 
tifiable  and  clearly  stated,  for  it  is  these  to 
which  benefit  is  attached.  If  this  linkage 
cannot  be  established,  the  discriminator 
should  be  reexamined.  These  are  issues 
relating  to  cost-benefit  analysis  proce¬ 
dure,  and  they  will  not  be  discus,sed  in 
detail  here.  The  principles  and  procedures 
employed  in  decision  analysis,  especially 


"A  set  of  dliscrimina* 
tors  must  be  estab¬ 
lished  that  validly 
characterises  the 
benefit  advantages 
eff  each  ef  the  pro¬ 
posals  over  ethers/' 
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those  of  Multi- Attribute  Utility  Analysis 
(usually  denoted  as  MAUA,  MAUT,  or 
MAU),  are  particularly  relevant  here. 

Treatment  of  Risk 

The  risks  associated  with  offeror  pro¬ 
posals  are  treated  to  a  degree  in  the  cost- 
realism  adjustments  described  in  the  first 
report  in  this  three-article  series.  The  most 
probable  cost  to  the  government  (MFC) 
thus  incorporates  some  of  the  proposal  risk 
implications.  However,  as  indicated  in  the 
first  report  in  this  series,  certain  discrimi¬ 
nators  will  also  involve  risks.  Avoidance 
of  risks  associated  with  one  offeror  can 
be  a  benefit  or  advantage  associated  with 
choosing  another  offeror’s  approach.  Thus 
analytic  procedures  for  quantifying  risks 
associated  with  offeror  proposals  are  very 
relevant.  These  will  not  be  discussed  here, 
but  it  is  quite  important  that  B  VWG  mem¬ 
bers  understand  and  can  employ  these  pro¬ 
cedures  or  can  get  assistance  in  doing  so. 

One  should  also  note  that  the  term 
“risk”  has  multiple  meanings  and  can  be 
a  source  of  confusion.  For  example,  some 


government  agencies  have  SSEB  (or  an 
equivalent  body)  members  evaluate  off¬ 
eror  proposals  by  assigning  each  factor  a 
rating  and  a  risk. 

Still  others  assign  factor  ratings  at  one 
level  and  risks  at  the  next  higher  level 
where  the  several  factor  ratings  are  aggre¬ 
gated  into  a  single  rating.  Usually  the  fac¬ 
tor  ratings  are  pegged  to  an  evaluation 
scale  or  standard.  The  risks,  if  defined,  are 
usually  briefly  described.  These  do  not 
represent  the  kind  of  risks  often  discussed 
in  economic  or  decision  analyses.  Yet  they 
do  represent  the  consensus  of  an  evalua¬ 
tion  panel  regarding  a  conditional  evalu¬ 
ation  of  an  aspect  of  an  offeror’s  proposal. 
Recall  that  it  is  not  these  risks  that  would 
be  directly  quantified,  but  rather  the 
judged  impacts  of  the  risks.  Clearly  the 
impact  must  be  carefully  characterized, 
and  the  approach  to  such  characterization 
must  be  consistent  throughout  the  analy¬ 
sis.  The  BVWG  should  carefully  plan  for 
this  difficult  process.  The  goal  is  not  for  a 
perfect  analysis,  but  it  is  to  achieve  a  con¬ 
sistent,  “reasoned”  analysis. 
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Endnote 


I.  In  one  version  of  this  approach,  ben¬ 
efits  are  characterized  as  “quantifiable” 
and  “nonquantifiable",  and  the  de¬ 
scribed  tradeoff  process  is  accom¬ 
plished  for  the  “quantifiable”  benefits. 
The  “nonquantifiable”  or  “qualified” 
benefits  can  only  be  used  as  tie-break¬ 
ers.  However,  the  di.stinction  between 


“quantified”  and  “nonquantified”  ben¬ 
efits  in  any  of  the  Best  Value  imple¬ 
mentations  is  not  clearly  defined.  In 
fact,  whether  a  benefit  can  be  quanti¬ 
fied  may  be  a  function  of  the  degree  of 
planning  for  such  tradeoffs  done  early 
in  the  acquisition  process.  The  term 
“quantification”  can  lead  to  confusion 
between  the  ability  to  quantify  and  the 
nature  of  the  scale  or  “uniqueness”  of 
the  measurements  obtained  from  the 
quantification  procedure  (Krantz,  et. 
al.,  1971). 
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OPINION 


COST  AS  AN  INDEPENDENT  VARIABLE: 
CONCEPTS  AND  RISKS 


Dr,  Beniamin  C.  Rush 


Cost  as  an  Independent  Variable  (CAIV),  implemented  in  early  1996,  is  a  new 
initiative  to  reduce  defense  system  costs.  Here  we’ll  look  at  the  definitions, 
concepts,  processes,  and  risks  of  CAIV,  with  examples  from  the  eight  flagship 
programs  that  are  leading  in  its  use. 


In  the  past  decade  of  tremendous 
changes  in  defense  systems  acquisi¬ 
tion,  the  most  significant  factor  is  the 
dramatic  drop  in  dollars  available  to  buy 
new  systems.  This  mandates  new  think¬ 
ing  on  strategies  and  processes  for  acqui¬ 
sition.  Part  of  this  change  in  thinking  is 
Cost  as  an  Independent  Variable  (CAIV), 
a  new  initiative  to  reduce  life-cycle  costs 
of  defense  systems.  CAIV  was  proposed 
in  1995  and  implemented  in  March  1996 
as  a  part  of  the  new  5000  Series  regula¬ 
tions  on  defense  systems  acquisition  (DoD 
5000. 2R,  1996).  Compliance  with  the 
principles  of  CAIV  is  required  for  all  ac¬ 
quisition  category  (ACAT)  I  and  lA  pro¬ 
grams  and,  at  the  discretion  of  the  com¬ 
ponent  acquisition  executive  (CAE),  the 
principles  may  be  applied  to  other  pro¬ 
grams.  Implementation  of  CAIV  is  basi¬ 
cally  in  two  steps:  first,  when  a  mission 
needs  statement  (MNS)  is  approved  (and 
the  concept  exploration  phase  begins),  an 
approach  is  laid  out  to  set  cost  objectives; 


and,  second,  upon  program  initiation  (usu¬ 
ally  at  Milestone  I  approval),  the  actual 
life-cycle  cost  objectives  are  established 
by  the  program  office. 

Two  Department  of  Defense  (DoD) 
working  groups  led  the  definition  and 
implementation  of  CAIV.  A  Defense 
Manufacturing  Council  working  group  de¬ 
veloped  a  CAIV  working  group  report  dis¬ 
seminated  in  December  1995  which  de¬ 
scribes  a  strategy  for  setting  aggressive, 
realistic  cost  objectives  for  acquiring  de¬ 
fense  systems  and  managing  the  associ¬ 
ated  risks.  In  June  1996,  the  Flagship  Pro¬ 
grams  Workshops  began  meeting  under 
the  leadership  of  Dr.  Spiros  Pallas  of  the 
Office  of  the  Secretary  of  Defense  (OSD). 
The  participants  include  representatives  of 
eight  defense  programs,  as  well  as  repre¬ 
sentatives  of  OSD,  the  Institute  for  De¬ 
fense  Analyses  (IDA)  and  the  Defense 
Systems  Management  College  (DSMC). 
Table  1  lists  the  eight  flagship  programs 
as  well  as  their  current  program  phase  and 
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a  short  description  of  the  weapon  system. 
Flagship  programs  are  sharing  problems 
and  solutions  in  implementing  CAIV 
policy. 

Definition 


CAIV  is  a  new  DoD  strategy  that  makes 
total  life-cycle  co.st  as  projected  within  the 
new  acquisition  environment  a  key  driver 
of  system  requirements,  performance 
characteristics,  and  schedules.  This  is  a 
180-degree  conceptual  change  in  thinking 
from  the  days  of  requirement-,  perfor¬ 
mance-,  and  sometimes  schedule-driven 
costs.  While  the  life-cycle  cost-perfor¬ 
mance-requirements  tradeoff  process  is 
the  heart  of  CAIV,  a  broader  definition  is 
necessary-  to  recognize  the  environment  in 
which  these  trades  take  place.  Programs 
are  being  aggressively  managed  to  meet 
program  objectives  concomitantly  with 
the  implementation  of  reform  initiatives 
such  as:  use  of  commercial  specifications 
and  practices;  use  of  integrated  product 
and  process  development  teams;  and  con¬ 
tractor  enterprise  reengineering.  Acquisi¬ 
tion  reform  initiatives  have  the  potential 
to  significantly  reduce  cost  and  change  the 
baseline  against  which  cost-performance- 
requirements  trades  are  benchmarked.  The 
Defense  Acquisition  Deskbook  provides 
a  description  of  CAIV  within  this  broader 


context;  “CAIV  is  a  strategy  that  entails 
setting  aggressive  yet  realistic  cost  objec¬ 
tives  when  defining  operational  require¬ 
ments,  acquiring  defense  systems,  and 
managing  achievement  of  these  objec¬ 
tives.  Co.st  objectives  must  balance  mis¬ 
sion  needs  with  projected  out-year  re¬ 
sources,  taking  into  account  existing  tech¬ 
nology,  maturation  of  new  technologies, 
and  anticipated  process  improvements  in 
both  DoD  and  industry”  (DoD,  1996).  In 
some  ways  CAIV  suffers  from  the  com¬ 
bination  of  too  many  initiatives  to  be  eas¬ 
ily  explained.  Philo.sophically  CAIV  is  the 
combination  of  all  the  best  practices  af¬ 
fecting  cost. 

Concepts 


The  implementation  of  CAIV  requires 
new  thinking  about  program  management. 
If  cost  is  truly  the  key  driver  of  perfor¬ 
mance  and  .schedule,  no  single  co.st  reduc¬ 
tion  strategy  is  likely  to  be  sufficient.  All 
cost  reduction  initiatives  must  be  consid¬ 
ered,  In  a  pre.sentation  by  the  Institute  for 
Defen.se  Analy.ses  at  the  Flagship  Work¬ 
shop  in  July  1996  (Bell,  1996),  a  hierar¬ 
chy  of  CAIV  co.st  levers  was  proposed. 
All  of  these  levers  are  important  in  CAIV 
implementation.  They  are  listed  in  rough 
order  of  potential  benefit  for  most  pro¬ 
grams; 
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PROGRAM 

PROGRAM 

DESCRIPTION 

PROGRAM 

STATUS 

EELV 

A  more  cost-effective  space 
launch  vehicle  for  medium  and 
heavy  lift  requirements 

Pre-EMD  start  Dec.  1996 

AIM-9X 

Next  generation  Sidewinder  air- 
to-air  missile 

EMD  start  Jan.  1997 

TACMS- 

Upgrade  of  tactical  ground-to- 

Currently  in  PDRR 

BAT  P3I 

ground  missile  -  new  seeker 

EMD  start  in  1998 

MIDS 

Third  generation  secure,  jam- 
resistant,  communications  system 
for  NATO  family 

EMD  contract  awarded  in  Mar 

1994 

Restructured  Jun.  1994 

CDR  in-process 

JASSM 

Long-range  air-to-surface  standoff 
missile 

Entered  2-year  competitive  PDRR 

CRUSADER 

155MM  self-propelled  Howitzer 
and  armored  resupply  vehicle 

Completion  of  PDRR  in  FY  2000 
Single  contractor  team 

JSF 

Advanced  Strike  Fighter  Aircraft 

Pre-PDRR 

SBIRS 

Space-based  infrared  surveillance 
system  for  missile  defense 

Entered  EMD  for  GEO  in  FY  1996 
PDRR  for  LEO  with  MS  II  in 

FY  1999 

Table  1 .  CAIV  Flagship  Pregrams 


1.  Requirements-cost-performance 
trades.  This  is  the  essence  of  CAIV 
and  is  discussed  in  detail  in  follow¬ 
ing  sections. 

2.  Acquisition  strategy.  Competition  is 
the  greatest  lever  that  the  govern¬ 
ment  has  in  the  early  stages  of  a  pro¬ 
gram  to  ensure  that  CAIV  objec¬ 
tives  are  met.  Because  of  this,  com¬ 
petition  should  be  maintained  as 
long  as  economically  practical. 


3.  Concurrent  engineering/integrated 
product  and  process  development 
(IPPD).  To  meet  an  aggressive  cost 
target,  team  members  must  cooper¬ 
ate  to  ensure  that  all  functional  plan¬ 
ning  be  integrated  and  that  difficul¬ 
ties  are  discovered  and  resolved 
early  on. 

4.  Contractor  enterprise  reengineering. 
The  lean  enterprise  philosophy  en¬ 
courages  industry  to  concentrate  on 
core  capabilities  and  to  develop 
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long-term  relationships  with  key 
suppliers  for  non-core  activities.  It 
also  requires  that  core  activities  be 
conducted  with  maximum  effi¬ 
ciency. 

5.  Commercial  specifications,  prac¬ 
tices.  and  components.  Acquisition 
reform  has  enabled  use  of  commer¬ 
cial  specifications  and  practices  in 
many  areas.  The  use  of  commercial 
components,  where  technically  fea¬ 
sible.  is  an  important  cost  reduction 
tool  for  many  programs. 

DoD  expects  cost  savings  from  these 
cost  levers  to  enable  50  percent  and  greater 
reductions  in  cost  from  the  old  way  of 
doing  business.  The  Joint  Direct  Attack 
Munitions  Program  is  a  frequently  cited 
example  of  a  program  that  is  achieving  this 
magnitude  of  reduction  from  the  broad 
impact  of  the  new  way  of  doing  busine.ss. 

The  preceding  consistently  addresses 
the  tradeoff  process  as  cost-performance 

and  require¬ 
ments.  This  em¬ 
phasizes  the  role 
of  the  user  and 
the  importance 
of  the  transition 
from  the  re¬ 
quirements  pro¬ 
cess  to  contract¬ 
ing  for  system 
performance 
goals.  The  process  considers  the  chang¬ 
ing  nature  of  requirements  as  system  de¬ 
velopment  progresses.  To  enhance  the  ef¬ 
fectiveness  of  CAIV,  programs  minimize 
the  number  of  system  perfonnance  param¬ 
eters  stated  in  the  Operational  Require¬ 
ments  Document  (ORD)  at  Milestone 


(MS)  I.  This  allows  performance  objec¬ 
tives  to  be  developed  that  are  achievable 
and  affordable  based  on  actual  develop¬ 
ment  and  additional  analysis  during  Pro¬ 
gram  Definition  and  Risk  Reduction 
(PDRR).  If  the  number  of  key  perfor¬ 
mance  parameters  are  kept  to  a  minimum 
while  continuing  to  meet  the  user’s  real 
needs,  greater  leeway  is  provided  for  fu¬ 
ture  tradeoffs.  The  system  performance 
parameters  called  out  in  the  ORD  are  des¬ 
ignated  key  performance  parameters  and 
are  not  tradable  below  a  threshold  value. 
Thus  for  key  performance  parameters  the 
only  trade  space  is  between  threshold  and 
objective  values.  Both  values  are  stated 
in  the  ORD  and  in  the  Acquisition  Pro¬ 
gram  Ba.selinc  (APB),  and  using  the  CAIV 
strategy  are  refined  until  MS  II. 

For  technical  performance  parameters, 
the  CAIV  targets  should  be  the  same  as 
those  in  the  APB.  For  CAIV  cost  thresh¬ 
old  and  objective  values,  there  are  poten¬ 
tial  problems  with  having  them  equiva¬ 
lent  to  the  APB  values.  The  program  bud¬ 
get  cannot  exceed  the  APB  cost  threshold 
and  the  cost  threshold  is  specified  as  10 
percent  above  the  objective  value 
(5000.2R,  Part  3.2.1  and  3.2.2.2).  This 
may  provide  little  cost  room  to  solve  tech¬ 
nical  performance  parameter  breaches. 

To  some  extent,  previous  attempts  at 
co.st-performance  trades  fell  victim  to  in¬ 
flexible  requirements  from  the  user  or 
overspecified  requirements  by  the 
acquirer.  Performance  goals  have  fre¬ 
quently  been  driven  by  available  technol¬ 
ogy  where  the  contractor  and  program 
management  office  (PMO)  strive  for  “the 
last  ounce  of  performance.”  The  thresh¬ 
old  and  objective  values  for  key  perfor¬ 
mance  parameters  are  initially  developed 
as  the  u.ser  translates  the  broadly  stated 


"To  some  extent, 
previous  attempts  at 
cost-performance 
trades  fell  victim  to 
Inflexible  require¬ 
ments  from  the  user 
or  overspecified 
requirements  by  the 
acquirer/' 
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mission  need  from  the  mission  area  analy¬ 
sis  into  a  system  description  for  the  ORD. 
An  analysis  of  alternative  system  concepts 
should  focus  on  determining  the  appro¬ 
priate  technical  performance  trades  prior 
to  the  initial  ORD  and  APB  at  Milestone  I. 
The  key  performance  parameters  are 
stated  in  the  initial  ORD  and  APB  and 
updated  at  each  milestone.  For  effective 
contracting,  performance  must  be  stated 
as  overall  system  performance  goals  rather 
than  as  detailed  specific  performance  pa¬ 
rameters.  Changing  these  goals  during 
system  development  because  of  changing 
mission  requirements  from  the  user  will 
greatly  hinder  the  CAIV  process.  Further, 
the  user  and  acquirer  must  be  willing  to 
accept  lesser  performance  for  less  cost 
within  the  trade  space.  Changing  the  cul¬ 
ture  regarding  lesser  but  acceptable  per¬ 
formance  is  critical  to  successful  imple¬ 
mentation  of  CAIV.  Thus,  the  user  must 
be  an  integral  player  throughout  the  pro¬ 
cess  as  the  cost-performance-require¬ 
ments  tradeoffs  are  made  in  each  phase  of 
the  life  cycle. 

Clearly  the  tradeoff  process  is  more  ef¬ 
fective  if  it  can  be  accomplished  earlier  in 
the  design  process.  A  large  percentage  of 
the  cost  is  determined  by  a  small  percent¬ 
age  of  the  design  decisions.  These  critical 
cost-driving  design  decisions  normally  are 
made  very  early  in  the  concept  selection 
and  design  process.  Because  of  this,  ex¬ 
pect  greater  success  in  implementing 
CAIV  for  programs  in  concept  explora¬ 
tion  or  program  definition  and  risk  reduc¬ 
tion  phases.  There  are  significant  problems 
estimating  production  and  operations  and 
support  (O&S)  costs  early  in  develop¬ 
ment  but  these  estimates  can  be  updated 
and  improved  over  the  product’s  life 
cycle.  Improved  estimates  will  have  the 


greatest  program  impact  if  competition 
continues. 

How  is  this  different  from  design-to- 
cost  (DTC)?  This  question  is  frequently 
asked  in  discussions  on  CAIV.  CAIV  em¬ 
bodies  more  than  the  tradeoff  process  that 
is  DTC  and  there  are  key  conceptual  dif¬ 
ferences.  Under  CAIV,  the  user  is  an  ac¬ 
tive  participant  in  the  tradeoff  process 
throughout  the  life  cycle.  This  is  not  nor¬ 
mally  the  case  with  DTC.  Another  key  dif¬ 
ference  is  CAIV’s  more  flexible  require¬ 
ment  based  on  threshold  mission  effective¬ 
ness.  Earlier  planning  in  the  life  cycle  by 
the  user  and  acquirer  with  an  iterative  re¬ 
fining  of  the  objectives  is  another  differ¬ 
ence.  In  the  past,  DTC  has  been  predomi¬ 
nantly  a  contractor’s  process,  executed 
during  the  system  design.  In  the  simplest 
terms,  consider  DTC  as  one  of  the  tools 
for  the  implementation  of  the  CAIV  con¬ 
cept. 

Processes 


The  DoD  initiative  on  IPPD  and  inte¬ 
grated  product  teams  (IPTs)  is  central  to 
the  implementation  of  CAIV.  Within  both 
contractor  and  government  organizations, 
it  is  expected  that  this  initiative  will  have 
been  implemented.  Under  the  direction  of 
the  government  program  manager  (PM), 
a  cost-performance  IPT  (CPIPT)  will  es¬ 
tablish  the  program  cost  objectives  and  fa¬ 
cilitate  the  cost-performance-require¬ 
ments  tradeoff  process.  Team  membership 
includes  the  user  from  the  outset,  and  con¬ 
tractor  representation  as  it  is  determined 
appropriate  (as  per  5000.2R,  Part  1,  Sec¬ 
tion  1.6).  Other  members  vary  depend¬ 
ing  on  the  phase  of  the  life  cycle  but 
could  include  the  service  cost  center  and 
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Figure  1.  Participants  in  the  CAIV  Process 


the  OSD  Cost  Analysis  Improvement 
Group  (CAIG)  as  does  the  Joint  Air-to- 
Surface  Standoff  Missle  (JASSM)  pro¬ 
gram.  A  detailed  discussion  of  the  mem¬ 
bership  and  roles  of  the  CPIPT  is  provided 
in  the  “Life  Cycle  Cost-Perfonnance  Con¬ 
cept  Paper”  (DoD,  1995). 

The  CAIV  process  is  an  iterative  one 
focused  on  the  PM  and  CPIPT  (Figure  1). 
The  PM  and  CPIPT  work  with  the 
Overarching-IPT  representing  the  pro¬ 
gram  evaluation  officer,  service  headquar¬ 
ters,  and  OSD  in  determining  funding, 
receiving  programmatic  direction,  and 
providing  program  status.  The  PM  and 
CPIPT  must  have  a  strong  working  rela¬ 
tionship  with  the  user  community  in  es¬ 
tablishing  cost-effective  requirements  and 
determining  priority.  The  PM  and  CPIPT 
have  a  number  of  supporting  acquisition 
organizations,  from  functional  support 


within  the  component  command  to  service 
cost  centers,  which  provide  cost  estimates 
and  analysis.  Design  and  cost  analysis  by 
the  contractors  provide  the  CPIPT  with  the 
information  necessary  to  analyze  cost- 
performance  tradeoffs.  This  circle  of  re¬ 
lationships  around  the  PM  and  CPIPT 
enable  a  sequence  of  activities  necessary 
to  accomplish  CAIV.  These  include  the  de¬ 
velopment  of  aggressive  cost  goals,  imple¬ 
mentation  of  incentives  to  encourage  the 
accomplishment  of  these  goals,  and  mea¬ 
surement  of  specific  CAIV  performance 
through  tracking  of  metrics. 

SiniNG  Aggressivi  Cost  Targets 

Developing  aggressive  cost  goals  re¬ 
quires  the  CPIPT  to  consider  a  number  of 
elements,  including  available  resources, 
costs  of  comparable  systems  and  compo¬ 
nents,  mission  effectiveness  studies,  tech- 
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nology  base  trends,  and  the  use  of  such 
initiatives  as  lean  manufacturing  and  com¬ 
mercial  business  practices.  The  CPIPT 
must  work  to  develop  initial  aggressive 
cost  goals  using  these  elements  and  the 
following  framework. 

1.  Using  affordability  as  the  key  cri¬ 
terion,  the  service  headquarters  di¬ 
vides  a  fixed  budget  among  com¬ 
peting  programs.  Here  the  cost  goals 
are  used  in  developing  budget  re¬ 
quired  for  that  program  and  com¬ 
pared  with  the  available  dollars  in 
the  Program  Objective  Memoran¬ 
dum  (POM)  years  based  on  the  pri¬ 
ority  level  established  by  the  ser¬ 
vice,  Joint  Requirements  Oversight 
Council  (JROC),  and  others.  This 
fixed  budget  based  on  the  priority 
of  the  program  is  the  reality  of  what 
is  available  for  structuring  the  pro¬ 
gram.  The  current  budget  may  be 
less  constraining  in  the  out  years, 
but  still  drive  the  program  acquisi¬ 
tion  strategy. 

2.  Using  mission  effectiveness  as  the 
key  criteria,  the  user  and  service 
headquarters  must  determine  “the 
most  bang  for  the  buck”  of  the  pro¬ 
posed  system.  Here  analytical  stud¬ 
ies  begin  with  mission  area  analy¬ 
sis  and  analysis  of  alternatives  and 
result  in  a  set  of  requirements  in  a 
mission  need  statement  and  the 
ORD.  This  analysis  would  look  at 
the  proposed  program  in  terms  of 
mission  effectiveness  versus  perfor¬ 
mance  requirements  and  perfor¬ 
mance  requirements  versus  cost. 
There  are  different  DoD  organiza¬ 
tional  elements  involved  in  this 


analysis,  depending  on  the  service: 
Center  for  Naval  Analyses  (Navy), 
Training  and  Doctrine  Command 
(TRADOC)  (Army),  combat  com¬ 
mand  (Air  Force),  and  Program 
Analysis  and  Evaluation  (PA&E) 
(OSD).  These  studies  provide  the 
necessary  tie  between  mission  re¬ 
quirements,  performance  param¬ 
eters,  and  the  cost  effectiveness  re¬ 
quired  of  the  system. 

3.  The  PMO  would  normally  have  ac¬ 
cess  to  independent  research  and 
contract  studies  by  contractors, 
which  provide  concepts  and  cost 
estimates  for  achieving  the  required 
system  performance  requirements. 
These  concepts  and  associated  costs 
may  vary  widely  from  one  study  to 
the  next  but  provide  the  critical  con¬ 
tractor  perspective  on  range  of  al¬ 
ternatives  and  provide  key  data  to 
the  above-mentioned  analysis  of 
alternatives  and  funding  exercises. 

The  PM,  through  the  CPIPT,  must  find 
a  set  of  initial  cost  goals  that  provide  an 
affordable  budget  and  still  enable  the  sys¬ 
tem  to  meet  at  least  the  threshold  require¬ 
ments  of  the  user.  If  the  cost  goals  include 
consideration  of  the  most  likely  cost  of  the 
performance  and  schedule  requirements, 
there  can  exist  a  legitimate  trade  space  for 
cost-performance  tradeoffs  and  the  cost 
targets  will  be  realistic.  If  initial  realistic 
cost  goals  cannot  be  developed  through 
this  trade  program  within  the  budget 
affordability,  the  program  is  not  viable. 
The  initial  cost  goals  will  be  refined  at 
each  stage  of  development  to  ensure  a 
balance  between  the  realistic  and  the  ag¬ 
gressive.  They  will  be  referred  to  as  cost 
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goals  by  Milestone  I.  as  cost  targets  by 
Milestone  II.  and  fimi  cost  targets  by  Mile¬ 
stone  III. 

The  key  focus  of  CAIV  is  on  the  total 
life-cycle  cost  (LCC)  of  a  program  with 
LCC  in  four  .separate  cost  objectives:  re¬ 
search.  development,  test,  and  evaluation 
(RDT&E):  production;  operations  and 
support;  and  disposal.  Here  we  give  pri- 
mar\’  attention  to  the  production  cost  ob¬ 
jective  and  the  operations  and  support  cost 
objective  reflecting  the  emphasis  of  the 
flagship  programs. 

The  production  cost  objective  is  defined 
in  several  ways.  The  basic  term  associ¬ 
ated  with  production  costs  of  individual 

items  of  a  pro¬ 
gram  is  known 
as  “as’erage  pro- 
curement  unit 
co,sts”  (APUC). 
The  APUC  is 
calculated  by 
dividing  the  to¬ 
tal  procurement 
cost  by  the  total 
procurement 
quantity  of  the 
program.  Also  of  interest  to  a  PM  is  the 
average  unit  cost  of  those  items  contracted 
for  in  each  production  lot.  The  average 
unit  production  cost  of  a  production  lot 
will  normally  vary  from  one  production 
lot  to  the  next  based  on  learning  curve 
theor)’  and  other  factors.  Further,  the  pro¬ 
duction  lot  average  unit  values  will  be  dif¬ 
ferent  from  the  APUC,  which  is  ba.sed  on 
the  total  program  quantity.  Additional  con¬ 
fusion  can  occur  when  one  compares  pro¬ 
duction  costs  of  different  programs,  be¬ 
cause  of  different  definitions.  Examples 
from  the  JASSM  and  AIM-9X  flag.ship 
programs  are  the  inclusion  in  production 


costs  of  “bumper  to  bumper”  warranty 
costs  (although  for  differing  periods). 
Other  programs  have  no  warranty  costs 
in  their  average  unit  costs. 

The  second  area  of  operations  and  sup¬ 
port  costs  is  even  more  difficult  to  pre¬ 
dict.  Contractually,  operations  and  support 
co.sts  may  best  be  handled  (as  several  of 
the  flagship  programs  have  done)  by  set¬ 
ting  aggressive  goals  for  key  performance 
parameters  that  drive  O&S  costs,  such  as 
mean  time  between  failure  (MTBF)  and 
mean  time  to  repair  (MTTR). 

Implementation  of  Incentives 

The  implementation  of  incentives  is  a 
critical  part  of  ensuring  the  necessary 
changes.  The.se  incentives  can  be  either 
positive  (achieving  targets)  or  negative 
(failure  to  meet  targets).  If  the  contractor 
is  not  meeting  the  program  cost  targets, 
an  acquisition  strategy  could  be  structured 
to  restart  competition.  An  acquisition  strat¬ 
egy  guide  provides  the  optimum  level  of 
competition  by  pha.se  is  one  of  the  most 
effective  ways  to  ensure  cost  is  minimized. 
Flagship  program  examples  are  the 
JASSM  and  Evolved  Expendable  Launch 
Vehicle  (EELV)  programs,  which  use  roll¬ 
ing  down-selects  (selecting  fewer  contrac¬ 
tors  for  each  succeeding  phase)  with  the 
final  development  contract  competition  in¬ 
cluding  low-rate  initial  production  and  the 
incentive  of  continuation  in  a  sole-source 
mode  as  long  as  the  final  cost  targets  struc¬ 
tured  during  the  final  competition  are  not 
breached. 

In  many  programs,  the  quantity  or  other 
factors  prevent  the  ability  to  have  compe¬ 
tition  in  production.  In  these  situations,  the 
use  of  award  or  incentive  profit  can  play 
a  major  role.  The  Crusader  program  is  an 
example  of  a  program  with  a  sole-source 
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contractor  in  development  through  pro¬ 
curement,  where  an  award  fee  is  being 
used  to  motivate  contractor  performance. 
This  is  in  an  environment  of  minimal  mil- 
specs,  mil-stds,  and  contract  data  require¬ 
ments  lists.  The  Space-Based  Infrared 
Systems  (SBIRS)  program  uses  an  incen¬ 
tive  fee  to  share  the  cost  savings  between 
the  government  and  the  contractor.  An 
important  motivator  for  all  programs  is  the 
shared  decision  role  through  contractor 
participation  on  the  CPIPT. 

Another  element  is  providing  appropri¬ 
ate  incentives  to  the  government  employ¬ 
ees  who  make  major  contributions  to  the 
success  of  the  program.  This  has  been  tried 
with  mixed  success.  One  of  the  major  dif¬ 
ficulties  is  that  monetary  awards  are  not 
allowed  for  military  members  of  the  gov¬ 
ernment  team,  but  a  change  in  the  law  is 
under  study. 

Measuring  Performance  Through 
Tracking  of  Metrics 

There  is  a  need  for  validated  cost  mod¬ 
els  to  track  life-cycle  cost  during  program 
Execution.  The  government  should  have 
access  to  contractors’  models  and  meth¬ 
odology.  This  does  not  mean  that  the  gov¬ 
ernment  and  contractor  have  the  same 
models,  but  they  work  together  to  share 
and  validate.  The  contractor’s  design-to- 
cost  system  must  provide  a  flow-down  of 
the  APUC  to  the  engineering  design  level 
with  status  reporting,  corrective  actions, 
and  trend  analysis.  The  reporting  process 
is  incorporated  into  the  contract  statement 
of  work.  The  Crusader  program  showed 
that  the  models  used  for  trades  were  inad¬ 
equate  for  cost  tracking.  The  AIM-9X  pro¬ 
gram  demonstrated  that  it  was  extremely 
valuable  to  establish  early  a  government- 
contractor  APUC  working  group.  An 


APUC  baseline  can  later  be  altered  to  ac¬ 
count  for  government-directed  design 
changes,  quantity  changes,  and  economic 
price  adjustments.  Any  change  in  the 
baseline  must  be  directly  traceable  so  that 
the  cause  and  magnitude  are  documented. 

The  operations  and  support  costs  track¬ 
ing  process  has  been  handled  by  the  flag¬ 
ship  programs  in  one  of  two  ways.  Where 
the  contractor  has  provided  a  warranty  as 
part  of  the  APUC,  the  government  need 
only  be  concerned  with  the  cost  models  at 
the  time  of  warranty  negotiation.  When 
there  is  no  warranty,  the  government  in¬ 
terest  shifts  to  the  impact  of  the  technical 
cost  drivers.  The  system  O&S  costs  are 
best  controlled  through  test  and  analysis 
of  the  technical  parameters  driving  O&S 
costs  such  MTBF  and  MTTR.  Technical 
performance  measurement  should  be  used 
to  track  all  critical  performance  param¬ 
eters  including  those  driving  O&S  costs. 

Risk  Management  in  CAIV 


The  areas  of  risk  listed  below  must  be 
addressed  as  a  part  of  the  CAIV  process. 
Some  of  these  risks  are  in  conflict  with 
others;  they  must  be  continually  balanced. 
The  process  is  an  iterative  one  and  the 
risks  come  into  play  multiple  times  dur¬ 
ing  the  life  of  the  program.  Among  the  key 
areas  that  CAIV  must  consider  are: 

1.  Risks  that  the  current  budget  and 
priority  decisions  for  a  system  are 
sufficiently  accurate  and  remain 
stable  over  the  program  life  cycle 
to  provide  realistic  system 
affordability. 
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The  program  budget  must  be  real¬ 
istic  and  stable  for  a  successful  pro¬ 
gram.  This  is  a  major  problem  in 
managing  most  acquisition  pro¬ 
grams:  it  will  be  even  more  critical 
under  CAIV,  where  cost  explicitly 
drives  performance  and  schedule. 

2.  Risks  that  the  threshold  perfor¬ 
mance  requirements  will  provide 
the  necessary’  mission  effectiveness 
and  will  be  stable  during  system  de¬ 
velopment  and  production,  and  risks 
that  the  difference  between  thresh¬ 
old  and  objective  requirements  will 
provide  sufficient  trade  space  to  al¬ 
low  tradeoffs  between  cost,  sched¬ 
ule,  and  perfomiance. 

The  balance  between  ensuring  that 
the  system  will  meet  the  user’s  true 
requirements  and  the  necessity  of 
the  threshold  requirement  being  .suf¬ 
ficiently  low  that  real  trade  space 
exists  between  the  threshold  and  ob¬ 
jective  is  critical  to  the  tradeoff  pro¬ 
cess. 

3.  Ri.sks  that  the  shape  of  the  function 
between  performance,  require¬ 
ments.  mission  effectiveness,  and 
cost  can  be  detemiined  and  utilized 
in  tradeoff  analysis. 

The  determination  of  this  function 
and  the  desire  to  find  the  “knee  of 
the  curve’’  will  require  not  only 
good  co.st  data  but  extensive  mod¬ 
eling  of  mission  effectivene.ss.  An 
excellent  example  is  the  work  of  the 
Joint  Strike  Fighter  program  in 
modeling  these  relationships. 


4.  Ri.sks  that  the  historical  database  for 
parametric  estimates  u.sed  in  cost 
effectiveness  as.sessment  is  suffi¬ 
ciently  applicable  to  the  system  be¬ 
ing  estimated  to  provide  an  accu¬ 
rate  most  likely  value  and  range  (or 
probability  distribution  function) 
for  the  costs  of  the  system. 

The  database  for  parametric  esti¬ 
mates  always  seems  populated  with 
programs  that  are  sufficiently  dif¬ 
ferent  in  technology,  design,  or  mis¬ 
sion  from  your  program  so  as  to  call 
into  question  the  validity  of  the  es¬ 
timate.  To  achieve  good  tradeoffs, 
one  must  have  good  cost  models 
with  valid  data  reflecting  the  cur¬ 
rent  acquisition  reform  initiatives. 

5.  Risks  that  the  interrelationships 
of  the  system  performance  require¬ 
ments  are  sufficiently  understood  to 
select  the  most  cost-effective  sys¬ 
tem  performance  objectives,  and 
risks  that  the  performance  require¬ 
ments  are  accurately  translated  to 
system  performance  contractual 
goals,  which  the  contractor  has  suf¬ 
ficient  incentive  to  achieve. 

The  system  performance  goals  are 
seldom  independent.  Understanding 
these  interrelationships  is  critical  to 
contracting  with  and  providing  in¬ 
centive  to  the  contractor. 

6.  Risks  that  the  contractor  DTC 
analyses  accurately  direct  the  sys¬ 
tem  performance  objectives  to  spe¬ 
cific  design  and  process  decisions, 
and  risks  that  the  contractor  detail 
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engineering  (bottoms-up)  cost  data 
at  the  design  level  is  sufficiently 
accurate  to  make  the  most  cost  ef¬ 
fective  design  trades. 

Does  the  contractor  have  a  good 
DTC  process?  Without  one, 
achievement  of  aggressive  cost  tar¬ 
gets  is  unlikely. 

7 .  Risks  that  technology  developments 
will  enable  the  achievement  of  spe¬ 
cific  design  and  process  goals. 

If  the  performance  requirements  are 
too  ambitious  and  can’t  be  achieved, 
the  cost  and  schedule  of  technology 
development  will  become  the 
drivers. 

The  central  feature  of  CAIV  is  the 
tradeoff  process;  determining  af¬ 
fordable  performance  and  schedul¬ 
ing  based  on  cost  goals  is  accom¬ 


plished  by  a  set  of  decisions  that 
balance  the  above  risks. 


Summary 


The  flagship  programs  will  demonstrate 
the  ability  of  the  CAIV  concept  to  achieve 
significant  savings.  Results  will  not  be 
available  for  some  time.  In  the  meantime, 
all  major  defense  acquisition  programs  in 
the  first  two  phases  of  the  life  cycle  are 
charged  with  implementing  this  concept 
and  were  required  to  submit  a  paper  on 
CAIV  implementation  by  July  1,  1996. 
These  programs  continue  to  report 
progress  on  this  concept  annually  to  their 
Milestone  Decision  Authority.  We  hope 
this  and  other  articles  on  the  implementa¬ 
tion  progress  of  CAIV  will  increase  un¬ 
derstanding  of  the  concepts  and,  by  so 
doing,  further  its  ability  to  succeed  as  a 
key  strategy  in  the  management  of  all  de¬ 
fense  system  acquisitions. 
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OPINION 


A  REVIEW  OF  THE  LITERATURE: 
COimPETITION  VERSUS  SOIE-SOURCE 
PROCUREMENTS 


William  N.  Washingloa 

Competitive  procurements  often  do  achieve  some  savings  over  sole-source 
procurements,  but  a  review  of  the  literature  analyzing  this  issue  shows  that 
the  choice  is  not  always  straightfonward.  The  savings  is  not  always  substantial, 
or  is  diminished  by  other  costs  associated  with  competition. 


On  what  criteria  should  the  decision 
to  pursue  competitive  or  sole- 
source  procurement  be  based?  A 
review  of  the  literature  brings  several 
points  to  the  fore.  First,  there  is  some  ra¬ 
tionale  for  supporting  competitive  over 
sole-source  procurements,  but  not  all  com¬ 
petitive  procurements  produce  savings; 
and  the  savings  associated  with  going 
competitive  are  far  less  than  the  25  per¬ 
cent  cited  by  former  Secretary  of  Defense 
Robert  S.  McNamara.  Next,  several  fac¬ 
tors  should  be  considered  prior  to  a  deci¬ 
sion  to  go  competitive,  such  as  produc¬ 
tion  quantity,  complexity  of  the  item,  ca¬ 
pacity  utilization  of  the  industry  involved, 
special  skills,  and  sufficient  data  on  the 
item.  In  addition,  a  cost-benefit  analysis 
should  probably  be  performed  to  deter¬ 
mine  the  possible  savings  as  a  result  of 
competition.  Further,  low  dollar  value 
spare  parts,  required  in  considerable  quan¬ 
tity,  or  component  parts  and  systems  that 


are  jointly  used  extensively  by  private  in¬ 
dustry,  would  seem  to  be  the  best  places 
to  implement  competitive  procurements. 

Older  Studies  on 

THE  Benefits  of  Competition 


The  previous  research  work  in  this  area 
has  seemed  to  follow  a  sequence,  from 
more-or-less  brief,  rapidly  compiled  stud¬ 
ies  to  more  detailed  and  objective  research 
over  time.  I  thus  reviewed  the  literature 
with  the  thought  not  only  to  compare  the 
contract  vehicles,  but  to  consider  the  evo¬ 
lution  of  the  studies,  placing  more  weight 
on  the  later  study  efforts.  To  begin  with, 
the  early  studies  were  generally  based 
upon  limited  sample  sizes,  and  dealt  pri¬ 
marily  with  small  systems  or  electronic 
components.  These  studies  generally 
found  consistent  cost  savings  associated 
with  competition  programs,  but  in  most 


173 


Acquisition  Review  Quarterly — Spring  1997 


instances  failed  to  take  into  account  all  the 
costs  associated  with  the  competition  pro¬ 
cess,  such  as  the  cost  of  conducting  the 
competition,  setup  costs  for  the  new  con¬ 
tractor,  special  tooling  and  government- 
furnished  equipment,  and  the  time  value 
of  money  to  set  up  the  new  contractor.  A 
brief  discussion  of  these  studies  follows. 

Carter  (1974)  proposed  that  the  Air 
Force  try  “directed  licensing,”  where  the 
original  contractor,  during  the  develop¬ 
ment  phase,  agrees  to  provide  rights  in  the 
data,  and  to  an  agreement  to  license  to 
whomever  the  government  designates  to 
produce  the  weapon  system  during  any  or 
all  production  runs  following  initial  pro¬ 
duction.  Carter  felt  this  procedure  would 
save  money  by  forcing  competition.  He 
stated  that  previous  contracting  studies 
showed  a  25  percent  reduction  in  cost  due 
to  competition. 

Olson,  Cunningham,  and  Wilkins 
(1974)  discovered  that  the  cost  savings 
associated  with  competition  of  spare  parts 
ranged  from  10  to  17  percent,  with  the 
most  likely  savings  being  1 2  percent.  They 
were  cognizant  of  competition  co.sts,  but 
felt  that  for  spare  parts  competitions,  they 
generally  would  be  negligible. 

In  a  larger  study,  Zusman  and  Asher 
(1974)  found  that  competition  reduced 
costs  by  an  average  of  37  percent.  How¬ 
ever,  as  mentioned  previously,  they  did  not 
take  into  account  the  costs  of  conducting 
the  competitions  or  their  associated  co,sts. 


Lovett  and  Norton  (1978)  compared 
price  behavior  on  11  competitive  contracts 
that  previously  had  been  sole  source.  They 
found  cost  savings  from  0  to  34  percent, 
but  they  also  did  not  take  into  account  the 
costs  associated  with  the  competition. 

Daly,  Gates,  and  Schuttinga  (1979)  ex¬ 
amined  31  programs  and  showed  an  aver¬ 
age  price  reduction  of  35  percent  for  com¬ 
petition  on  five  missiles,  a  bomb,  a  guid¬ 
ance  unit,  and  assorted  electronic  compo¬ 
nents.  They  speculated  that  savings  for  a 
“split  award”  would  be  about  a  10  per¬ 
cent  reduction,  and  a  20  percent  reduction 
for  winner-take-all  competitions.  This 
study  failed  to  consider  competition  costs. 

Drinnon  and  Hiller  (1979)  expanded 
upon  the  work  of  Lovett  and  Norton  by 
reviewing  45  additional  programs.  They 
also  found  savings  reductions  ranging 
from  -1 6  percent  to  67.7  percent,  with  the 
median  around  39  percent.  Like  the  pre¬ 
vious  studies,  most  of  the  items  were  sub- 
assemblies  and  small  electronic  compo¬ 
nents.  Major  systems  in  their  study  only 
achieved  10-18  percent  reductions  (i.e., 
foreard  area  alerting  radar  [FAAR],  tube 
launched  optically  tracked  wire  guided 
missle  [TOW]  and  Shillelagh)  DEFINE. 
They  likewise  did  not  take  into  account 
the  costs  associated  with  competition. 

Kratz  and  Cox  (1982)  expanded  upon 
the  conceptual  framework  of  Drinnon  and 
Hiller,  and  suggested  that  what  transpired 
with  the  creation  of  competition  was  a 
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shift  and  rotation  of  the  learning  curve, 
with  an  immediate  drop  in  the  first  unit 
cost  and  a  steeper  learning  curve.  In  ap¬ 
plying  their  approach  to  five  missile  pro¬ 
curements,  they  found  that  the  first  unit 
cost  was  reduced  by  between  14  percent 
(4  percent  shift  and  8  percent  rotation)  to 
46  percent  (14  percent  shift  and  13  per¬ 
cent  rotation).  The  model  outlined  in  this 
approach  is  available  from  the  Defense 
Systems  Management  College  under  the 
name  of  the  Competition  Evaluation 
Model  (CEM),  version  2.0  (1992). 
Beltramo  (1989),  however,  took  exception 
with  the  logic  behind  this  model,  and  in  a 
study  performed  for  the  Naval  Center  for 
Cost  Analysis  found  only  one  example  out 
of  six  cases  where  there  was  a  shift  and 
rotation.  In  the  remaining  cases,  he  found 
a  downward  initial  shift  with  an  upward 
rotation  (i.e.,  a  lower  price  for  the  first 
competitive  lot,  followed  by  a  flatter  learn¬ 
ing  curve  than  expected  for  the  sole 
source). 

It  is  hard  to  determine  from  these  early 
studies  how  beneficial  competition  is  to 
the  procurement  process,  since  they  do  not 
take  into  account  the  costs  associated  with 
conducting  the  competition,  and  the  stud¬ 


ies  vary  considerably  in  terms  of  consis¬ 
tency  of  results  from  one  study  to  the  next, 
for  the  same  data.  However,  it  does  seem 
obvious  that  there  is  a  general  cost  sav¬ 
ings  associated  with  competition,  espe¬ 
cially  on  spare  parts. 

In  an  attempt  to  compare  the  studies’ 
results  for  this  problem  of  consistency,  I 
reviewed  the  same  procurements  in  dif¬ 
ferent  studies  (see  Table  1).  In  these  sys¬ 
tems  (i.e.,  data  points),  there  was  consid¬ 
erable  variability  in  results  from  one  study 
to  the  next — where  there  should  have  been 
substantial  similarity.  For  instance,  some 
studies  described  a  procurement  as  hav¬ 
ing  produced  a  cost  savings;  other  studies 
pronounced  the  same  procurement  to  have 
caused  a  loss.  Most  striking  were  results 
for  the  Sidewinder  9D/G  and  the  Sparrow 
7F  competitions,  which  were  significantly 
different  in  different  studies;  other  systems 
showed  considerable  variation  from  one 
study  to  the  next.  This  variability  can  be 
attributed  to  the  studies’  use  of  different 
definitions,  and,  as  a  result,  different  costs 
were  applied  from  one  study  to  the  next. 
Hampton  (1984)  presents  a  good  example 
of  how  this  occurs.  On  one  system,  the 
Shillelagh,  he  shows  that  savings  can  vary 


Table  1 .  Variqiice  Among  Studies  on  Cost  Savings 


STUDIES 

ZUSMAN 

LOVETT 

DALY 

DRINNON 

KRATZ 

GREER 

SYSTEMS 

Range 

TOW 

48 

9 

9 

12 

20 

26 

40 

SHILLELAGH 

0 

6 

-8 

9 

- 

-5 

17 

BULLPUP 

14 

- 

32 

27 

46 

18 

32 

SIDEWINDER  9D/G 

- 

- 

-5 

1 

- 

-71 

72 

SIDEWINDER  9B 

- 

- 

1 

-6 

17 

- 

23 

SPARROW  7F 

- 

- 

- 

- 

14 

-25 

39 
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from  -14  percent  to  22  percent  depend¬ 
ing  upon  the  data  used,  statistical  meth¬ 
ods,  and  definition  of  what  constitutes  sav¬ 
ings.  The  foregoing  studies  show  that  it  is 
hard  to  place  a  firm  number  on  the  actual 
savings  associated  with  competition. 

Second  Phase  of  Studies  on  the 
Benefits  of  Competition 


Either  as  a  result  of  the  previous  stud¬ 
ies.  or  perhaps  relating  to  increasing  pub¬ 
lic  interest  in  reducing  defense  cost.s,  the 
DoD  Cost  Analysis  Symposium  of  1982 
generated  four  papers  on  the  topic  of  com¬ 
petition.  These  papers  attempted  to  pro¬ 
vide  a  more  comprehensive  approach  to 
the  question  of  savings  resulting  from 
competition;  and  also  took  a  slightly  dif¬ 
ferent  approach  to  research  in  this  area, 
discussing  several  constraints  that  should 
be  considered  prior  to  a  competition  deci¬ 
sion. 

Trainor’s  review  (1982)  of  Lovett  and 
Norton  and  of  Daly,  Gates,  and  Schuttinga 
found  that  the  majority  of  items  (48/55) 
compared  in  these  studies  were  nonmajor 
systems  with  unit  costs  of  between  $4,100 
and  $8,400  (fiscal  year  1 980  dollars).  The 
only  major  weapons  systems  in  these  stud¬ 
ies  were  one  ship,  one  medium-size  mis¬ 
sile.  and  one  small  helicopter.  He  suggests 
that  the  re.sults  concerning  the  benefits  of 
competition  should  only  be  applied  to 
nonmajor  system  procurements.  In  addi¬ 
tion,  Trainor  gives  several  reasons  why 
competition  may  not  either  be  practical  or 
produce  cost  savings  in  the  future,  espe¬ 
cially  if  current  trends  for  defense  contrac¬ 
tors  continue.  These  points  are  rather  in¬ 
teresting.  and  in  light  of  our  current  de¬ 
fense  draw-down,  will  be  discussed  later. 


Watkins  (1982)  followed  up  on  the 
Kratz  and  Cox  model  for  estimating  the 
slope  for  competitive  contracts.  He  dis¬ 
cussed  the  historical  data  by  commodity 
area  (e.g.,  electronics,  missiles)  and  what 
the  rotation  and  shifts  could  be  for  them 
based  upon  previous  contracts.  He  also 
proposed  the  use  of  “should  costs,”  using 
the  model  to  determine  the  learning  curve 
that  the  contractor  should  agree  to  for  pro¬ 
duction. 

Smith  and  Lowe  (1982),  like  Watkins 
(1982),  looked  at  the  Kratz  and  Cox 
(1982)  model  for  estimating  the  slope  dif¬ 
ferences  between  competitive  and  sole- 
source  procurements.  Their  results  sup¬ 
ported  the  shift  and  rotation  premise  and 
suggested  that  between  a  15  and  25  per¬ 
cent  savings  on  spare  parts  could  be 
achieved  by  competition.  They  did  not 
mention  whether  the  cost  of  the  competi¬ 
tion  was  taken  into  account. 

Carrick  (1982)  discussed  experience 
curVes  and  the  factors  that  influence  them. 
Like  Trainor  (1982),  he  mentioned  that 
contractors  have  several  problems  in  their 
estimation  process  for  competitive  bids 
that  may  cause  cost  growth  over  the  ini¬ 
tial  estimate.  For  instance,  in  the  DIVAD 
program  the  winning  contractor  had  not 
even  generated  designs  for  several  of  the 
equipments,  yet  submitted  a  cost  estimate 
for  them.  Also,  in  the  Viper  and  Copper¬ 
head  programs,  neither  of  the  winning 
contractors  adequately  understood  the 
technology  underlying  their  designs,  much 
less  the  exceptional  difficulties  in  defin¬ 
ing  and  implementing  a  high  rate  of  pro¬ 
duction  technology.  These  examples  point 
out  that  one  cannot  just  use  the  bid  price 
from  the  contract  as  data  for  competition 
studies;  the  actual  production  costs  should 
be  used. 
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More  Comprehensive  Studies  on 
Competition _ 

Following  this  period,  the  research  on 
sole-source  versus  competition  changed 
from  somewhat  simple  comparisons  to 
multiple  factor  analyses.  These  studies 
recognized  that  there  were  several  possible 
factors  that  could  come  into  play  in  af¬ 
fecting  the  costs  associated  with  contract¬ 
ing.  A  number  of  these  well-done  studies 
were  master’s  theses  from  the  Air  Force 
Institute  of  Technology. 

Brost  (1982)  conducted  a  regression 
approach  to  determine  the  savings  asso¬ 
ciated  with  competition,  comparing  the 
estimated  sole-source  cost  on  spare  parts 
procurements  to  the  actual  competition 
prices,  controlling  for  inflation  and  com¬ 
modity  type.  His  results  ran  counter  to  the 
earlier  spare  parts  studies,  and  indicated  a 
general  negative  trend  as  a  result  of  com¬ 
petition.  These  results  could  have  been 
influenced  by  the  small  number  of  pro¬ 
curements  that  met  his  criteria  for  inclu¬ 
sion  into  the  study  (36).  Further,  while 
recognizing  that  there  were  additional 
costs  associated  with  competition,  the 
study  did  not  add  these  costs  to  the  com¬ 
petition  side  of  the  equation;  had  they 
been,  the  results  of  this  analysis  would  be 
even  less  favorable  toward  competition. 

Zamparelli  (1983)  followed  up  on  this 
spares  analysis,  and,  in  turn,  found  some 
savings  associated  with  competition  (4.1 
to  11.2  percent).  But  in  several  instances 
competition  was  not  found  to  be  benefi¬ 
cial.  For  example,  where  relatively  few 
companies  can  supply  a  particular  aircraft 
engine’s  spare  parts,  even  if  proprietary 
data  are  not  involved,  competition  was  not 
effective  in  reducing  costs,  since  the  sec¬ 
ond  source  of  supply  may  need  to  retool 


and  change  its  machine  specifications  in 
order  to  produce  the  parts.  In  addition, 
when  the  spare  parts  exceeded  $1,000  in 
unit  costs,  competition  did  not  save 
money.  Finally,  there  were  some  instances 
in  which  competition  increased  costs  by 
two  to  eight  times  the  sole-source  cost — 
but  these  instances  may  stem  from  the  part 
not  being  manufactured  any  longer.  The 
study,  like  the  previous  studies,  did  not 
consider  the  cost  of  competition  in  its 
analysis. 

Greer  and  Liao  (1983)  investigated  con¬ 
tractor  profitability  and  capacity  utiliza¬ 
tion  in  relation  to  competition  cost  sav¬ 
ings.  Using  three  of  the  six  missile  com¬ 
petitions  from  Kratz  and  Cox  (1982),  they 
concluded  that  competition  produces 
greater  savings 
when  firms  are 
at  low  capacity. 

But  when  ca¬ 
pacity  utiliza¬ 
tion  was  high, 
there  was  little 
benefit  attrib¬ 
uted  to  competi¬ 
tion.  The  worst 
cases  occurred 
when  capacity  utilization  was  above  80  per¬ 
cent.  In  those  instances,  there  were  net  losses 
associated  with  competition. 

Heinz  (1983)  looked  at  a  factorial  ap¬ 
proach  to  sole  source  versus  competition. 
He  suggested  that  for  the  early  develop¬ 
ment  of  armament  systems  sole  sourcing 
was  best,  but,  as  the  systems  matured  to 
the  6.5  level,  competition  became  more 
favorable.  His  suggestions  seemed  to  prin¬ 
cipally  be  related  to  the  complexity  of  the 
system,  in  that  the  more  complex  the  pro¬ 
cess,  the  more  appropriate  sole  source 
became. 


"The  worst  coses 
occurred  when 
capacity  utilization 
was  above  80  per¬ 
cent.  In  those  in¬ 
stances,  there  were 
net  losses  associated 
with  competition." 
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Hampton  ( 1 984)  produced  an  excellent 
paper  that  reviews  the  above-mentioned 
studies,  critiquing  them  upon  their  meth¬ 
odology  and  suggesting  a  more  appropri¬ 
ate  approach  to  determine  if  competition 
was  worthwhile.  Generally,  he  came  to  the 
conclusion  that  competition  was  not  al¬ 
ways  cost  effective  or  practical,  and  that 
in  order  to  determine  if  there  were  any  ad¬ 
vantages  to  a  sy.stem  going  competitive,  a 
cost-benefit  analysis  should  be  performed 
that  would  take  into  account  all  the  co.sts 

associated  with 


Gansler  discusses 
hew  the  "fair  and 
open  environment" 
that  Congress  has 
created  can  lead  to 
toe  many  bidders 
entering  the  cempe< 
titien— more  than  is 
good  for  either  the 
government  or  for 
the  contractors 
themselves. 


the  government 
and  the  contrac¬ 
tors,  and  would 
use  discounted 
dollars  in  accor¬ 
dance  with 
0MB  Circular 
A-94.  His  paper 
was  basically 
broken  into 
three  sections. 
The  first  section 


was  a  complete 
discussion  of  previous  research,  then  the 
factors  that  should  be  considered  in  de¬ 


termining  if  competition  were  cost  effec¬ 
tive.  and,  third,  a  discussion  of  a  cost-ben¬ 


efit  approach  that  could  be  applied  to  de- 
tennine  the  reasonableness  of  competition. 
He  also  discussed,  in  the  second  section, 
several  studies  that  took  these  additional 


government  and  contractor  co.sts  into  ac¬ 
count.  and  found  that  competition  was  not 
cost  effective  for  those  systems. 

Gable  (1985)  looked  at  whether  com¬ 
petition  reduced  spare  parts  procurement 
costs.  The  study  did  indicate  a  savings 
associated  with  competition,  but  he  states 
that  competition  is  not  always  possible  for 
several  reasons  (e.g..  inadequate  or  miss¬ 


ing  data,  proprietary  rights,  shrinking  in¬ 
dustrial  base).  He  recognized  also  that 
there  are  several  costs  associated  with 
competition  that  might  outweigh  the  ben¬ 
efits  in  gross  savings  (competition  person¬ 
nel  costs,  contracting  personnel  co.sts,  in¬ 
creased  processing  time  required  to  con¬ 
duct  the  competition,  and  the  additional 
paperwork  required). 

Presar  (1986)  di.scus.sed  how  the  pres¬ 
sure  to  increase  competition  would  cause 
increasing  workload  requirements  on  the 
commodity  commands,  in  terms  of  per¬ 
sonnel  and  time  to  conduct  these  procure¬ 
ments.  These  manpower  requirements 
would  be  borne  by  the  commodity  com¬ 
mands  and  would  not  be  funded  by  the 
weapons  systems  nor  out  of  the  normal 
command’s  budget,  thus  causing  the  of¬ 
fices  in  those  commands  to  absorb  the  in¬ 
creased  man-hours  out  of  their  existing 
workforce. 

Berg,  Dennis,  and  Jondrow  ( 1 986)  per¬ 
formed  a  literature  review  of  the  previous 
studies  on  sole-source  versus  competitive 
procurement.  They  recognized  the  incon¬ 
sistencies  of  the  previous  studies  and  at¬ 
tempted  to  outline  why  differences  may 
have  occurred  (e.g.,  use  of  differing  data, 
different  adju.stments,  different  assump¬ 
tions).  Their  recognition  of  the.se  possible 
problem  areas  and  the  subsequent  effect 
upon  the  previous  studies  was  quite  good. 
They  also  suggested  that  the  price  improve¬ 
ment  curve  model  of  Kratz  and  Cox  ( 1 982) 
may  not  take  enough  variables  into  consid¬ 
eration  for  tme  forecasting  purpo.ses. 

In  his  book  Affording  Defense,  Gansler 
(1989)  discus.ses  how  the  “fair  and  open 
environment”  that  Congress  has  created 
can  lead  to  too  many  bidders  entering  the 
competition — more  than  is  good  for  either 
the  government  or  for  the  contractors 
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themselves.  He  describes  one  case  in 
which  DoD  spent  time  and  money  evalu¬ 
ating  fifty  bids  for  a  few-hundred-dollar 
item.  Situations  like  these  hardly  make 
sense,  and  can  promote  inexperienced, 
weak  manufacturers,  when  DoD  with  its 
substantial  buying  power  should  be  ob¬ 
taining  the  most  effective  weapons  for  the 
lowest  cost.  He  also  stresses  the  impor¬ 
tance  of  continuous  competition,  where, 
if  possible,  not  only  the  initial  procure¬ 
ment  is  competed  but  also  the  production 
contracts;  preferably  with  a  leader-fol¬ 
lower  award,  so  that  there  continues  to  be 
a  competitive  pressure  on  manufacturers. 
He  emphasizes,  however,  that  competition 
should  make  sense,  and  that,  in  an  envi¬ 
ronment  stressing  competition  and  low 
cost,  the  quality  of  DoD  items  could  be 
threatened  if  it  is  carried  too  far.  He  sums 
up  these  concepts  with  the  following  state¬ 
ment:  “Competition  for  its  own  sake  is 
clearly  wrong;  however,  when  competi¬ 
tion  makes  good  management  sense  and 
when  best  value  is  emphasized,  that  is  a 
different  story.” 

Kitfield  (1989)  discusses  whether  some 
programs  represented  as  competitive  were 
really  so.  He  also  describes  a  Navy  study 
of  eight  separate  weapons  systems  that 
estimated  the  cost  of  bringing  on  a  sec¬ 
ond  source  at  2-4  percent  of  the  total  cost 
of  the  procurement. 

Boger,  Greer,  and  Liao  (1990)  assert 
that  competition  in  weapons  systems  pro¬ 
curement  does  not  always  produce  sav¬ 
ings.  They  reemphasize  Greer  and  Liao’s 
(1983)  previous  study,  where  capacity  uti¬ 
lization  above  80  percent  produced  losses 
when  systems  were  competed.  They  also 
discussed  several  factors  that  could  make 
competition  less  effective  than  in  private 
industry;  when  the  government  is  the  sole 


buyer,  with  only  limited  production  and 
few  companies  capable  of  producing  the 
items,  the  government  is  required  to  help 
establish  the  second  source. 

Flynn  and  Herrin  (1990)  show  that  the 
Navy  has  been  having  success  with  com¬ 
petitive  procurements  on  large  weapons 
systems,  achieving  a  14  percent  savings 
(these  savings  did  not  take  all  competi¬ 
tion  costs  into  account,  however).  They 
estimate  that  the  startup  costs  for  the  sec¬ 
ond  source  represented  2.4  percent  of  the 
total  program  costs.  But  they  temper  the 
14  percent  estimate  by  saying  that  the  pre¬ 
vious  procurements  occurred  during  the 
1980s  defense  buildup,  and  may  not  hold 
in  the  current  defense  drawdown  period 
with  reduced  quantities. 

Carlson,  Hamre,  and  McNicol  (1990) 
discussed  several  issues  concerning  weap¬ 
ons  system  competitions  at  the  DoD  Cost 
Analysis  Symposium  (1989).  This  was  the 
second  time  that  a  majority  of  symposium 
papers  dealt  with  competition  (1982).  The 
authors  covered  several  areas  of  possible 
concern  for  fu¬ 


ture  competi¬ 
tion,  such  as 
system  com¬ 
plexity  and 
whether  com¬ 
plexity  itself 
would  preclude 
dual  sourcing. 
They  also  as¬ 
serted  that  dual 


"...the  current 
preoccupation  with 
price  is  not  in  keep¬ 
ing  with  the  new 
trends  In  total 
quality  manage¬ 
ment,  and  that  best 
value  should  be  the 
principal  goal  for 
defense  procure¬ 
ments.." 


sourcing  may 

be  driving  companies  to  share  less  infor¬ 
mation  with  one  another,  out  of  fear  that 
they  may  end  up  competing  at  a  later  date, 
and  that  this  impaired  the  technical  capa¬ 
bility  associated  with  new  defense  tech¬ 
nologies.  They  also  stated  that  the  current 
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preoccupation  with  price  is  not  in  keep¬ 
ing  with  the  new  trends  in  total  quality 
management,  and  that  best  value  should 
be  the  principal  goal  for  defense  procure¬ 
ments. 

Elliot  (1990)  reviewed  the  impact  of 
competition  on  the  quality  of  the  items 
procured.  The  study  found  no  significant 
difference  in  quality  as  a  result  of  chang¬ 
ing  from  a  sole-source  to  a  competitive 
producer.  However,  these  procurements 
were  for  spare  parts  and  may  not  be  rep¬ 
resentative  of  major  systems  or  compo¬ 
nents  under  development. 

Wandland  and  Wickman  (1993)  found, 
as  have  authors  of  previous  studies,  that 
competition  resulted  in  reduced  costs  over 
sole-source  procurements,  though  the  dif¬ 
ference  was  not  statistically  significant. 
The  study  also  examined  the  que.stion  of 
whether  contractors  might  be  buying  in 
on  competitive  contracts.  Here  they  found 
that  counter  to  expectations,  competitive 

contracts  had 
less  cost  and 
schedule 
growth  than 
sole-source 
contracts, 
though  the  dif¬ 
ferences  were 
not  statistically 
significant. 
Like  previous 
studies,  the  costs  associated  with  compe¬ 
tition  were  not  considered  in  their  result.s, 
though  the}'  were  aware  of  several  com¬ 
petition  costs  (i.e.,  technology  transfer  to 
second  source,  additional  government 
management,  time  value  of  money,  pur¬ 
chasing  reprocure-ment  data,  special  tool¬ 
ing  and  test  equipment). 


Discussion 


Given  the  variability  of  results  from  the 
preceding  studies,  and  the  subsequent  rec¬ 
ognition  that  several  factors  are  involved 
in  the  ultimate  determination  of  whether 
competition  is  cost  effective,  it  seems  pru¬ 
dent  to  take  a  conservative  approach  to  the 
question  of  when  competition  should  be 
used.  Like  several  other  investigators  in 
this  area,  I  have  come  to  the  conclusion 
that  competition  “saving.s”  are  dependent 
upon  .several  factors,  ranging  from  indus¬ 
trial  base  issues  to  how  costs  are  defined 
in  the  analysis.  Trainor  (1982)  and  Gable 
(198.S)  discussed  several  industrial  ba.se 
issues  that  could  influence  production 
co.sts,  and  that  should  be  considered  when 
deciding  whether  to  use  sole  sources  or 
competition  to  procure  an  item. 

One  of  these  issues  is  production  rate. 
In  single-line  production  (where  only  one 
type  of  item  can  be  produced  on  the  pro¬ 
duction  line),  higher  production  rates  al¬ 
low  more  efficient  production,  and  so, 
lower  costs.  This  factor  was  coming  into 
play  in  1982,  with  decreasing  production 
rates,  and  has  continued  to  be  a  factor  as 
weapon  systems  have  become  more  com¬ 
plex,  and  require  higher  sophistication 
than  standard  manufacturing  products. 

The  stable  production  rate  is  another 
important  factor.  In  single-line  production, 
a  stable  production  rate  allows  for  more 
efficient  production,  and  so,  lower  co.sts. 
Stable  production  rates  were  becoming  a 
problem  for  military  manufacturers  in 
1 982,  and  have  continued  to  be  a  factor  as 
funding  for  military  programs  has  under¬ 
gone  continuing  readjustments,  which  in 
turn  causes  production  slippages. 

Production  quantity  is  a  combination 
of  the  previous  two  factors.  In  single-line 
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production,  large  quantities  allow  more 
efficient  production,  and  so,  lower  costs. 
This  factor  had  been  decreasing  for  10 
years  prior  to  1982,  and  continues  to  de¬ 
crease  in  the  present  environment. 

Time  required  to  stabilize  design  is 
another  element  to  consider.  Unless  the  de¬ 
sign  is  firm,  there  is  the  possibility  of  cost 
growth.  The  increased  complexity  and 
testing  requirements  of  weapon  systems 
back  in  1982  prompted  this  concern, 
which  has  continued  to  increase  with  the 
current  sophistication  and  complexity  of 
state-of-the-art  systems.  Some  examples 
from  Garrick’s  study  (1982)  were  the 
DIVAD  program,  in  which  the  winning 
contractor  had  not  even  designed  several 
of  the  components  when  it  submitted  the 
bid.  In  the  Viper  and  Copperhead  pro¬ 
grams,  neither  of  the  winning  contractors 
adequately  understood  the  technology  or 
the  high  rate  production  techniques  re¬ 
quired  when  they  bid  on  these  systems. 

Capacity  utilization  (in  terms  of  both 
workers  and  facilities)  is  another  issue  that 
affects  savings.  As  a  company’s  plant  uti¬ 
lization  increases,  the  associated  costs  for 
its  product  decrease  because  of  a  reduc¬ 
tion  in  overhead  and  excess  capacity.  This 
point  was  also  recognized  by  Greer  and 
Liao  (1983)  and  Boger,  Greer,  and  Liao 
(1990):  When  capacity  utilization  exceeds 
80  percent,  the  superiority  of  competition 
over  sole-source  acquisition  begins  to  di¬ 
minish,  perhaps  because  companies’  effi¬ 
ciencies  then  operate  at  about  the  same 
level,  and  their  costs  are  similar.  Defense 
contractors  continue  to  merge,  and  most 
are  now  operating  at  or  near  full  capacity; 
they  cannot  achieve  significant  savings  by 
reducing  excess  overhead. 

Special  production  skills  and  facili¬ 
ties  also  affect  the  sole  source  versus  com¬ 


petition  decision.  It  is  easier  to  establish  a 
second  production  source  if  the  need  for 
specialized  skills  and  facilities  does  not 
exist.  However,  with  weapon  systems  be¬ 
coming  increasingly  unique,  only  limited 
facilities  are  available  to  produce  some 
systems  (e.g.,  tanks,  submarines,  aircraft 
carriers),  so  that  the  pool  of  competitors 
for  an  increasing  number  of  weapon  sys¬ 
tems  is  reduced.  Zamparelli  (1983)  found 
that  relatively  few  companies  could  sup¬ 
ply  the  components  to  manufacture  air¬ 
craft  engine  parts. 

It  is  difficult  to  establish  a  second  pro¬ 
duction  source  if  production  drawings 
are  not  available.  As  funding  has  become 
tighter  over  the  years,  several  programs 
have  opted  for  reducing  the  number  of 
system  drawings  for  their  components,  or 
not  updating  those  drawings  as  design  modi¬ 
fications  have  changed  the  components. 

Proprietary  data  rights  also  affect  the 
sole-source  versus  competition  decision. 
It  is  difficult  to  establish  a  second  source  if 
the  system  or  component  uses  proprietary 
information.  Many  contractors  incorporate 
components  and  parts  in  their  systems  for 
which  they  hold  the  proprietary  rights. 

In  addition,  several  costs  associated 
with  competition  must  be  taken  into  ac¬ 
count  to  determine  if  competition  will  re¬ 
ally  save  money.  Hampton  (1984)  and 
Beltramo  (1990)  discuss  several  of  these 
in  detail: 

•  the  source  selection  costs,  which  in¬ 
cludes  both  the  government  person¬ 
nel  and  facilities  required,  along 
with  the  contractor’s  cost  to  develop 
the  proposal; 

•  second  source  development  costs, 
such  as  updating  the  technical  data 
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package,  special  tooling  and  test 
equipment  required,  cost  of  trans¬ 
ferring  the  technical  data  to  the  new 
source,  and  first  article  testing; 

•  other  possible  liabilities  to  the 
government,  concerning  the  unde¬ 
preciated  assets  that  the  government 
may  have  to  pay  for,  or  fumi.sh  to 
the  new  source; 

•  quantity  and  learning  curve  losses 
in  production,  if  quantities  are  split 
between  several  sources; 

•  increased  contract  administration 
costs,  if  quantities  are  split  between 
several  sources; 

•  increased  technical  data  administra¬ 
tion  cost  for  updating  more  than  one 
source;  and 

•  company-funded  research  and  de¬ 
velopment  costs  that  need  to  be  re¬ 
captured  by  the  original  developer. 

Added  to  these  are  the  logistics  costs 
associated  with  maintaining  multiple  ver¬ 
sions  of  a  system  in  the  inventory,  and  the 
required  spare  parts  unique  to  each  ver¬ 
sion.  These  costs  have  not  been  discussed 
in  the  literature.  Tied  into  this  is  the  in¬ 
creased  training  required  for  repair  of  the 
different  versions,  and  their  respective 
technical  manuals. 

To  thoroughly  cover  the  topic,  I  dis¬ 
cussed  the.se  issues  with  the  leading  pro¬ 
fessors  and  experts  in  the  field:  Beltramo 
( 1 996),  Fullerton  ( 1 995),  McAfee  ( 1 995), 
Rao  (1995),  Rogerson  (1995),  Vincent 
(1995),  Wilson  (1995),  and  Yao  (1995). 
From  these  discussions  I  learned  of  addi¬ 


tional  studies  and  books  the  initial  litera¬ 
ture  searches  did  not  uncover. 

Anton  and  Yao  (1987,  1989,  1990,  and 
1992)  have  published  several  theoretical 
papers  on  the  effects  of  full  costing  knowl¬ 
edge  versus  incomplete  information  on  the 
bidding  process.  They  point  out  that  the 
developer’s  production  experience  pro¬ 
vides  a  cost  advantage  over  a  second- 
source  bidder,  but  this  pricing  advantage 
can  be  offset  if  is  not  opened  to  competi¬ 
tion  until  later,  and  the  initial  cost  infor¬ 
mation  is  provided  to  all  bidders.  They 
also  note  that  in  an  environment  with  un¬ 
equal  cost  information,  both  the  bidder  and 
the  buyer  benefit  from  a  split  award  over 
a  winner-take-all  award;  in  an  open  cost 
knowledge  environment,  the  buyer  re¬ 
ceives  a  lower  cost  under  a  winner-take- 
all  process. 

Recently,  Fullerton  (1995a  and  1995b), 
Fullerton  and  McAfee  (1996),  and  Taylor 
(1995)  have  expres.sed  some  novel  and 
interesting  proposals  concerning  compe¬ 
tition.  These  are  termed  “research  tourna¬ 
ments,”  in  which  the  competition  proce¬ 
dure  is  structured  as  an  auction  and  proto¬ 
type  competition,  with  the  winner  awarded 
a  “prize”  for  the  best  product.  The  auc¬ 
tion  component  consists  of  the  participants 
paying  a  fee  for  entering  the  tournament, 
which  could  be  pooled  across  the  partici¬ 
pants  to  defray  the  cost  of  the  prize,  or 
offset  the  cost  of  conducting  the  competi¬ 
tion.  This  prize  could  either  be  a  set 
amount  of  money  based  upon  what  the 
government  determined  the  work  effort  to 
be  worth,  or,  if  the  contract  award  was 
large  enough  or  had  commercial  applica¬ 
tions,  the  award  would  merely  be  the  win¬ 
ning  of  the  contract,  since  the  follow-on 
work  would  generate  sufficient  commer¬ 
cial  incentive. 
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An  article  on  optimal  procurement 
mechanisms  by  Manelli  and  Vincent 
(1995)  is  similar  to  the  work  of  Anton  and 
Yao  above  (1987,  1989,  1990,  and  1992) 
and  looked  at  theoretical  competitions.  He 
proposed  that  the  optimal  competition 
environment  would  be  first  to  offer  to  a 
select  group  of  companies,  in  a  sequen¬ 
tial  process,  a  fixed  price  to  perform  the 
work,  and  if  they  all  should  reject  the  price, 
then  hold  an  auction. 


Conclusions _ 

This  review  of  the  competition-versus- 
sole-source  procurements  literature  makes 
apparent  the  following  points.  First,  there 
is  probably  some  rationale  supporting 
competitive  over  sole-source  procure¬ 
ments,  but  not  all  competitive  procure¬ 
ments  produce  savings;  and  the  savings 
are  probably  far  less  than  25  percent.  Next, 
one  should  consider  several  factors  before 
a  competitive  procurement  is  chosen; 
these  include  production  quantity,  com¬ 
plexity  of  the  item,  capacity  utilization  of 
the  industry  involved,  special  skills,  and 
sufficient  data  on  the  item.  In  addition, 
decision  makers  should  probably  perform 
a  cost-benefit  analysis  before  choosing 
competitive  procurement,  to  determine  if 
that  avenue  will  actually  result  in  any  sav¬ 
ings.  Lastly,  competition  is  probably  the 
best  choice  for  acquisition  of  low-dollar- 
value  spare  parts  required  in  considerable 
quantity,  or  for  component  parts  and  sys¬ 
tems  that  are  jointly  and  extensively  used 
by  private  industry. 

Currently,  competition  is  the  prescribed 
means  of  procurement,  but  we  should  be 
aware  of  its  ramifications  both  for  private 
industry  and  the  military.  The  current  hy¬ 


persensitivity  of  private  industry  concern¬ 
ing  research  and  manufacturing  technolo¬ 
gies  (apparent  from  companies’  great  con¬ 
cern  about  sharing  information  with  other 
contractors,  for  fear  that  they  may  be  com¬ 
peting  in  the  future)  is  one  result  of  this 
policy.  Carlson  (Carlson,  Hamre,  and 
McNicol,  1990)  discussed  this  effect  with 
Hamre  and  McNicol  at  the  1989  Depart¬ 
ment  of  Defense  Cost  Analysis  Sympo¬ 
sium,  and  pointed  out  that  in  the  past, 
when  specific  companies  had  “baronies” 
for  a  particular  area,  they  maintained  top- 
notch  engineers  for  long  periods  of  time 
at  one  location.  These  groups  of  experts 
generated  a  research  synergy  that  led  to 
the  development  of  new  technologies  and 
a  willingness  to  share  technical  informa¬ 
tion  with  other  industries.  Carlson’s  state¬ 
ment  that  this  situation  no  longer  exists 
has  recently  been  echoed  by  industry  rep¬ 
resentatives  who  describe  today’s  environ¬ 
ment  as  “kill  or  be  killed”  (National  De¬ 
fense,  1996).  Companies  hide  what  they 
are  doing,  and  do  not  allow  their  employ¬ 
ees  to  discuss  their  work  at  symposiums 
like  the  American  Defense  Preparedness 
Association,  or  the  National  Industrial  Se¬ 
curity  Association  meetings.  This,  in  turn, 
he  stressed,  handicaps  development  of 
new  technologies  in  these  defense  indus¬ 
tries,  and  drives  the  armed  services  to  de¬ 
pend  more  on  commercial  developments 
to  generate  the  high-technology  equip¬ 
ment  required  to  maintain  an  edge  over 
other  countries.  Hamre  pointed  out  the 
current  overemphasis  on  cost  cutting, 
which  runs  contrary  to  principles  of  total 
quality  management,  where  price  is  not 
the  primary  issue.  This  point  of  view  has 
caused  a  change  in  recent  years  to  “best 
value”  competitions,  where  quality  and 
value  are  considered  in  relation  to  price. 
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TUTORIAL 


SEIECTING  EFFEQIVE  ACQUISITION 
PROCESS  METRICS 


Aron  Pinker,  Charles  G»  Smith,  andJark  IV.  Booher 

Metrics  for  assessing  the  acquisition  reform  process  are  now  being  actively 
sought  by  the  DoD.  It  is  difficult  to  identify  meaningful  metrics  that  can  be 
conveniently  calculated.  Using  our  experience  from  the  Partnership  Process 
for  Electronic  Warfare  (EW)  Acquisition,  we  describe  a  reasonable  approach 
to  effective  selection  of  metrics.  We  examine  DoD  initiatives  aimed  at  measuring 
acquisition  reform,  identify  a  process  for  establishing  metrics,  suggest  a  basis 
for  ordering  metrics,  and  provide  examples  of  metrics. 


This  article  is  a  result  of  the  Secretary 
of  the  Air  Force  Electronic  Combat 
Division’s  (SAF/AQPE’s)  effort  to 
design  a  new  approach  to  the  acquisition 
of  Electronic  Warfare  (EW)  systems.  SAF/ 
AQPE  assembled  an  EW  Acquisition  Part¬ 
nership  team  to  design  an  acquisition  pro¬ 
cess  that  seamlessly  integrates  the 
warfighter’s  requirements  with  product 
development  and  testing.  From  its  incep¬ 
tion,  the  EW  team  recognized  that  improv¬ 
ing  the  EW  systems  acquisition  process 
requires  identification  of  the  baseline  ac¬ 
quisition  process  for  EW  systems  and 
definition,  or  development,  of  a  new  ac¬ 
quisition  process.  To  attain  this  objective 
and  demonstrate  that  improvement  has 
been  achieved,  it  is  imperative  to  have 
some  measures,  or  metrics,  for  compar¬ 
ing  the  old  process  (baseline)  with  the  new 
process  (acquisition  reform).  Here  we 
present  some  of  our  insights  on  metrics 


that  could  be  useful  to  the  DoD  acquisi¬ 
tion  community. 


Problems  in  Developing  Metrics 


Most  people  who  work  with  metrics 
recognize  that  it  is  not  easy  to  identify 
meaningful  metrics  that  can  be  conve¬ 
niently  calcu¬ 


lated  (Dellinger, 
1994). 

The  main  con¬ 
sideration  in  Air 
Force  acquisi¬ 
tion  reform  is 
whether  the 
new  process  en¬ 
ables  us  to  field 
better  weapon 
systems, 
and  cheaper. 


"Metrics  allow  us  to 
baseline  vrhere  we 
are,  identify  the 
impediments  to  the 
process,  and  track 
the  impact  of  man¬ 
agement  artiens  on 
processes  and  ether 
process  changes." 

-Gen.  Thomas  R. 

Ferguson,  Jr. 
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The  first  problem  with  developing  metrics 
for  the  acquisition  process  is  that  we  can¬ 
not  directly  measure  these  attributes.  So 
they  are  useless  as  metrics;  we  must  use 
other,  quantifiable,  “surrogate”  metrics 
instead.  But  it  is  not  easy  to  decide  what 
these  surrogate  metrics  should  be,  and  it 
is  not  always  clear  how  they  would  con¬ 
tribute  to  the  goal  of  fielding  military  sys¬ 
tems  that  are  better,  faster,  and  cheaper. 

A  second  problem  with  formulating 
metrics  is  the  fact  that  a  weapon  system 
acquisition  takes  place  over  a  long  period 
of  time.  The  success  or  failure  of  the  ac¬ 
quisition  is  determined  in  retro.spect  by 
how  well  the  weapon  system  has  served 
the  military.  Consequently,  we  can  a.ssess 
the  success  of  the  acquisition  process  only 
in  a  post  mortem.  Such  an  assessment 
would,  of  course,  be  of  merely  historical 
interest  and  little  practical  use.  We  will 
later  suggest  a  method  for  creating  a  top- 
down  (or  bottom-up)  hierarchy  of  metrics 
that  links  surrogate  metrics  to  the  true 
metrics  by  means  of  Quality  Function 
Deployment  (QFD)  (Fortuna,  1988)  and 
the  Analytic  Hierarchy  Process  (AHP) 
(Saaty,  1980). 


Devilopihg  Standardized  Metrics 

As  the  defense  acquisition  system  is 
being  streamlined,  DoD  is  also  consider¬ 
ing  ways  to  measure  the  improvement  as 
it  occurs.  Measuring  improvement  starts 
with  identifying  the  changes  in  process 
brought  about  by  acquisition  reforni  and 
providing  a  comprehensive  plan  for  es¬ 
timating  and  measuring  these  changes. 
Dr.  Paul  Kaminski,  Under  Secretary  of 
Defense  (Acquisition  and  Technology)  be¬ 
lieves  that  the  Pentagon  should  have  De¬ 
fense  Department-wide  metrics  (Mead¬ 
ows,  1995).  If  this  standardization  is 
achieved,  it  would  provide  a  u.seful  basis 
for  comparing  the  various  acquisition  re¬ 
form  initiatives. 

This  raises  the  question  of  which 
metrics  can  be  shared  by  all  commands 
and  which  would  only  apply  to  special¬ 
ized  activities.  For  example,  while  the  EW 
acquisition  community  may  have  some 
metrics  that  are  shared  by  the  general  DoD 
procurement  community,  EW  may  have 
some  unique  service-specific  or  area-spe¬ 
cific  metrics.  Additionally,  if  different 
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Table  1.  Initial  Metrics 


TYPE  OF  METRIC 

METRIC 

Cost 

Consumable  item  price  index,  miiitary 
specification  conversion  price  benefit 

Acquisition  performance 

Contract  defaults,  contract  changes 

Schedule 

Acquisition  phase  time,  administrative 
lead  time,  multiyear  procurements; 

FACNET  transactions,  logistics 
response  time 

Commercial  practices 

Contract  specifications,  credit  card 
purchases 

commands  have  different  senses  of  mis¬ 
sion  criticality,  they  would  weight  the 
shared  metrics  differently. 

Consequently,  using  our  experience 
from  the  EW  Acquisition  Partnership,  we 
intend  to  describe  a  reasonable  approach 
for  selecting  metrics.  In  the  following  sec¬ 
tions  we  will  examine  DoD  initiatives 
aimed  at  measuring  acquisition  reform, 
identify  a  process  for  establishing  metrics, 
provide  examples  of  metrics,  and  suggest 
a  basis  for  selecting  and  ordering  metrics. 

Recent  DoD  Initiatives 


Process  Action  Teams  (PATs) 

Last  year  Dr.  Paul  Kaminski  and  Col¬ 
leen  A.  Preston,  former  Deputy  Under 
Secretary  of  Defense  for  Acquisition  Re¬ 
form,  chartered  several  process  action 
teams  (PATs)  to  recommend  actions  for 
reforming  DoD  acquisition  practices  and 
to  define  metrics  for  assessing  the  effec¬ 
tiveness  of  the  recommended  reforms.  The 
PATs  were  fairly  successful  in  identifying 


simplifications  and  improvements  to  re¬ 
form  DoD  acquisition  practices.  The  defi¬ 
nition  of  metrics,  however,  has  turned  out 
to  be  a  major  difficulty.  The  PATs  have 
struggled  to  come  up  with  at  least  some 
metrics.  Yet,  they  never  explained  the  in¬ 
terrelationship  and  connection  of  these 
metrics  to  the  over-all  goal. 

The  Tiger  Team 

After  the  PATs’  attempts  at  defining 
metrics,  the  Defense  Standards  Improve¬ 
ment  Council  formed  a  metrics  Tiger 
Team,  led  by  the  Office  of  the  Assistant 
Secretary  of  the  Army,  Research,  Devel¬ 
opment,  and  Acquisition  (OASA/RDA), 
Acquisition  Reform  Office,  to  develop 
metrics  and  a  method  for  collecting  data  for 
these  metrics.  This  team  has  proposed  a  set 
of  initial  strategic  outcome  metrics  for  mea¬ 
suring  the  impact  of  acquisition  reform. 
Preston  has  approved  the  strategic  outcome 
metrics  in  Table  1  and  has  authorized  the 
OASA/RDA  to  collect  the  necessary  data. 

It  appears  that  the  Tiger  Team  selected 
these  metrics  because  they  are  relatively 
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"Because  surrogate 
metrics  are  net  true 
metrics,  we  need  to 
knew  hew  strongly 
they  represent  the 
true  metrics." 


easy  to  collect.  From  the  warfighters’  per¬ 
spective,  the  categor)'  of  “system  perfor¬ 
mance”  has 
been  omitted. 
Also,  the  Tiger 
Team  has  not 
addressed  the 
issue  of  quick 
integration  of 
advanced  tech¬ 
nologies.  The  categories  of  metrics  in 
Table  1  will  probably  be  expanded  in  the 
future. 

Appendix  A  presents  a  list  of  the  stra¬ 
tegic  outcome  metrics  that  have  been  sug¬ 
gested  to  the  Acquisition  Reform  Senior 
Steering  Group.  Appendix  B  pre.sents  the 
algorithms  for  computing  the  initial  set  of 
selected  metrics. 


AQUismoM  Riform  Benchmarking  Group 

On  Sept.  18,  1995,  Preston  established 
the  Acquisition  Reform  Benchmarking 
Group  (ARBG),  chaired  by  William  E. 
Mounts  from  her  office.  The  ARBG  will 
receive,  assemble,  and  as.sess  data  from 
these  and  other  acquisition  reform  strate¬ 
gic  outcome  metrics.  The  group  will  also 
as.sess  the  suitability  of  other  metrics  pro¬ 
posed  by  the  various  acquisition  reform 
PATs.  Interim  results  can  be  found  on  the 
World  Wide  Web  at: 

http;/Avww.acq.osd.mil/ar 


Conceptual  Considerations  in 
Selecting  Metrics 


in  general  as  shown  in  Table  1 . 

Acquisition  reform  metrics  are  the  nu¬ 
merical  values  by  which  we  gauge 
progress  toward  meeting  acquisition  re¬ 
form  objectives. 

If  the  overall  objective  of  the  acquisi¬ 
tion  reform  is  to  field  faster,  better,  and 
cheaper  weapon  systems,  then  a  true  met¬ 
ric  would  be  any  numerical  value  that  en¬ 
ables  us  to  assess  how  much  faster,  how 
much  better,  and  how  much  cheaper  a 
given  acquisition  process  is.  Unfortu¬ 
nately,  we  do  not  have  such  true  metrics; 
we  do  not  know  how  to  directly  measure 
these  qualities.  The  terms  faster,  better, 
and  cheaper  have  so  many  possible  mean¬ 
ings  that  we  must  restrict  these  terms  to 
some  of  their  more  specific  characteris¬ 
tics.  To  do  this  we  have  to  use  “surrogate 
metrics.” 

A  surrogate  metric  is  a  measurable 
characteristic  of  the  acquisition  process 
that  presumably  reflects  the  behavior  of  a 
true  metric. 

Because  surrogate  metrics  are  not  true 
metrics,  we  need  to  know  how  strongly 
they  represent  the  true  metrics.  Moreover, 
some  metrics  may  be  better  described  as 
submetrics  that  together  constitute  a 
higher  level.  This  grouping  leads  to  a  hi¬ 
erarchical  structure  of  metrics  with  the 
many  surrogate  metrics  at  the  bottom  and 
a  few  true  metrics  at  the  top.  This  group¬ 
ing  also  requires  us  to  determine  how  the 
lower-level  metrics  contribute  to  the 
higher-level  metrics. 


Defining  True  and  Surrogah  Metrics 

Because  our  intere.st  is  in  the  acquisi¬ 
tion  process,  we  have  chosen  to  define 
metrics  for  this  process  rather  than  metrics 


Brainstorming  Potential  Metrics 

One  can  usually  gather  many  potential 
metrics  for  a  process.  Follow  these  guide- 
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lines  to  brainstorm  for  potential  metrics: 

1.  Identify  the  specific  segment  of  the 
process  that  is  to  be  evaluated. 

2.  Identify  the  pertinent  properties  of 
what  is  to  be  measured. 

3.  Identify  types  of  potential  metrics. 

4.  Select  a  few  metrics  and  provide  a 
rationale  for  the  specific  selection. 

5.  Find  bounds  on  what  is  being  mea¬ 
sured. 


Avoidihg  Ineffective  Metrics _ 

Once  you  have  discovered  several  po¬ 
tential  metrics,  determine  which  ones  will 
be  most  useful.  A  good  metric  will  be 
meaningful,  logical,  simple  to  express, 
understandable,  repeatedly  and  quickly 
derivable,  unambiguously  defined,  and 
derivable  from  economically  collectible 
data. 

In  addition,  a  good  metric  will  indicate 
trends,  suggest  corrective  actions,  and 
numerically  describe  the  progress  toward 
the  objective. 

While  it  is  important  to  be  able  to  iden¬ 
tify  a  good  metric,  it  is  also  important  to 
know  what  is  not  a  metric.  Metrics  are  not 
charts,  schedules,  goals,  objectives,  strat¬ 
egies,  plans,  missions,  guiding  principles, 
counts  of  activity,  single-point  statistics, 
or  rankings.  Also,  tracking  a  process  is  not 
necessarily  the  same  as  tracking  a  metric. 
In  spite  of  this,  one  IPX  suggested  using 
the  following  measurements  as  metrics: 


“Program  managers  should  track  use 
of  military  unique  specifications  and 
standards  and  report  out  at  milestone/ 
program  reviews”  (OUST[A&T], 
1994,  p. 53) 

“The  Standards  Improvement  Ex¬ 
ecutives  shall  be  responsible  for 
tracking  implementation  of  all  acqui¬ 
sition  reform  issues  related  to  speci¬ 
fications  and  standards”  (OUSD 
[A&T],  1994,  p.  165). 

Another  IPX  suggested  that  contractor 
responses  to  a  questionnaire  would  serve 
as  an  input  to  a  database,  which  would 
eventually  be  used  for  developing  metrics. 
This  proposed  questionnaire  included  the 
following  questions  (OUSD[A&X],  1994, 
p.  27): 

1.  Are  there  any  military  specifications 
or  standards  required  as  a  part  of  this  so¬ 
licitation  which  could  be  better  served  by 
a  commercial  specification? 

2.  Were  any  changes  required  in  your 
routine  manufacturing  process  specifically 
to  accommodate  this  DoD  purchase?  Do 
you  believe  that  the  changes  added  value 
to  the  product? 

3.  Did  you  offer  alternatives  to  require¬ 
ments  of  any  military  specifications  or 
standards?  Do  you  feel  that  your  alterna¬ 
tives  were  given  adequate  consideration  by 
the  procuring  agency?  Were  any  adopted? 

4.  How  would  you  improve  the  solici¬ 
tation  to  allow  you,  and  other  contractors, 
to  quote  a  lower  product  cost  while  main¬ 
taining  identical  product  performance? 
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Such  questionnaires  may  solicit  ideas 
for  reform,  but  they  seem  to  have  little 
value  for  forming  metrics,  because  they 
do  not  call  for  numerical,  quantifiable  re¬ 
sponses.  Could  questionnaires  be  used  for 
developing  metrics?  Could  they  be  used 
to  provide  metrics  that  are  of  immediate 
use?  The  answers  to  these  questions  are 
not  easy  and  may  depend  on  the  particu¬ 
lar  program  for  which  metrics  arc  devel¬ 
oped. 

The  Required  Details  for  a  Metric 

The  definition  of  a  metric  tends  to  be 
simple,  because  a  metric  should  be  easy 
to  explain  and  calculate.  Yet,  from  the 
technical  point  of  view,  many  details  of 
each  metric  must  be  specified  to  ensure 
commonality  of  the  derived  metrics.  For 
each  metric,  at  least,  the  following  details 
must  be  specified: 

1.  Description  of  the  population  that 
the  metric  includes. 

2.  Identification  of  the  source  of  data. 

3.  Precise  definition  of  key  terms. 

4.  Statement  of  the  mathematical  ex¬ 
pressions  that  will  be  used  to  derive  vari¬ 
ous  values. 

5.  Specification  of  frequency  of  mea¬ 
surements  to  derive  the  metric. 

6.  Description  of  the  graphics  that  will 
be  used  to  display  the  data. 

7.  Specification  of  user’s  tolerance  lev¬ 
els  (i.e.,  “control  limits”). 


8.  Listing  of  desired  outcomes  ex¬ 
pressed  in  terms  of  a  positive  or  negative 
trend  (not  a  numerical  goal). 

9.  Linkage  between  the  metric  and  the 
activity  being  measured. 

1 0.  Linkage  between  the  surrogate  met¬ 
ric  and  the  tme  metric. 


Creating  a  Metric 


Having  laid  out  general  guidelines  and 
requirements  for  designing  metrics,  we 
now  describe  a  step-by-step  procedure  for 
establishing  and  using  metrics  to  assess 
improvement  in  the  acquisition  process. 
Follow  this  procedure  to  create  a  metric: 

Identify  the  purpose  of  the  metric. 

The  purpose  of  the  metric  should  reflect 
the  purpose  of  the  acquisition  reform  ini¬ 
tiative  and  its  mission,  vision,  goals,  and 
objectives. 

Develop  an  operational  definition  of 
the  metric.  Define  the  who,  what,  when, 
why,  and  how  of  this  metric  in  sufficient 
detail  to  permit  consistent,  repeatable,  and 
valid  measurement  of  the  acquisition  pro¬ 
cess. 

Examine  exi.sting  means  of  measur¬ 
ing.  Check  whether  existing  metrics  or 
process  measuring  means  could  be 
adapted  to  satisfy  the  operational  defini¬ 
tion  of  the  metric.  In  other  words,  do  not 
“reinvent  the  wheel.” 

Generate  new  metrics.  In  the  past, 
most  metrics  were  not  process-oriented; 
they  were  usually  related  to  final  outputs. 
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products,  or  services.  The  focus  is  now  on 
improving  the  new  acquisition  process  so 
that  superior  final  outputs  are  obtained. 
Currently,  the  underlying  assumption  for 
generating  metrics  is  that  by  monitoring 
changes  in  the  process  we  can  assess  pro¬ 
cess  improvements. 

Conduct  a  “goodness  of  fit”  check. 
Check  whether  the  newly  generated  met¬ 
ric  satisfies  the  previously  stated  attributes 
of  a  good  metric.  Make  sure  that  all  the 
previously  stated  details  can  be  provided 
for  this  metric.  Check  objectivity  of  the 
metric  to  ensure  that  the  measurements  or 
observations  do  not  affect  the  outcome. 

Choose  a  mode  of  display.  Decide  on 
the  mode  for  presenting  the  metric.  This 
decision  will  affect  data  collection  and 
availability. 

Conduct  a  “sanity”  check.  Acquire 
data  for  deriving  the  metric.  Derive  the 
metric  for  various  instances  and  ask  the 
customer  to  judge  whether  the  metrics  are 
meaningful.  Does  the  metric  measure 
what  it  is  supposed  to  measure?  Do  the 
metric  values  correspond  to  intuition?  If 
the  answer  is  uncertain,  return  to  the  sec¬ 
ond  step. 

Form  a  consensus.  Obtain  consensus 
or  buy-in  from  participants. 

Create  a  database.  Collect  and  ana¬ 
lyze  the  metric’s  data  over  time  and  for  dif¬ 
ferent  cases.  Examine  trends.  Can  you  ad¬ 
equately  explain  counterintuitive  metric 
values?  For  what  lengths  of  time  does  the 
metric  stabilize  (i.e.,  does  not  deviate  sig¬ 
nificantly  from  its  mean)? 


Communicate  the  metric.  Be  open  to 
constructive  criticism.  Be  ready  to  make 
adjustments. 

Employ  the  metric.  Metrics  are  just  a 
means  to  an  end — continuous  process 
improvement.  If  there  is  confidence  in  the 
metric,  then  it  should  be  used;  otherwise, 
look  for  a  new  metric.  Employing  the  met¬ 
ric  allows  you  to  refine  it  and  make  it  an 
even  better  tool.  (AFSC,  1991). 

Examples  of  Metrics 


The  following  illustrate  metrics  at  vari¬ 
ous  levels  of  abstraction  and  areas  of  in¬ 
terest  to  the  acquisition  process.  These 
metrics  were  collected  from  various 
sources,  but  most  of  them  fall  into  the  fol¬ 
lowing  categories:  cost,  acquisition  per¬ 
formance,  schedule,  commercial  practices, 
weapon  system  performance,  and  technol¬ 
ogy  innovation. 

Program  office  overhead.  Program 
overhead  as  a  fraction  of  total  program 
cost. 

Specifications  conversion.  Number  of 
military  specifications  that  have  been  re¬ 
placed  with  industry  standards. 

Specifications  elimination:  Number  of 
military  specifications  that  have  been 
eliminated;  or,  reduction  in  number  of 
specifications  and  standards  specified  in 
a  contract. 

Cost  and  pricing  data.  Percentage  of 
competitive,  negotiated  procurements  re¬ 
quiring  certified  cost  and  pricing  data;  or. 
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ratio  of  the  number  of  contract  awards 
with  cost  and  pricing  data  to  the  total  num¬ 
ber  of  contract  awards. 

Funding  stability.  The  number  of  times 
a  program  changes  in  terms  of  quantity  or 
cost,  due  to  fiscal  pres.sures  external  to 
program  executive  officers  (or  an  equiva¬ 
lent  management  level). 

Program  cost.  Change  in  program  cost 
as  a  consequence  of  changed  acquisition 
processes. 

Unit  production  price.  Change  in  unit 
production  cost  as  a  consequence  of 
changed  acquisition  processes. 

Unit  life-cycle  cost.  Change  in  pro¬ 
jected  unit  life-cycle  cost  as  a  consequence 
of  changed  acquisition  processes. 

Operational  performance  versus 
cost.  Compare  operational  test  results  ver¬ 
sus  specified  performance  for  accuracy 
and  reliability  with  Average  Unit  Produc¬ 
tion  Price  Milestone  I  cost  analysis  im¬ 
provement  group  estimates  versus  con¬ 
tractor  production  proposals. 

Commercial  practices.  Compare  busi- 
ness-as-usual  versus  commercial  practices 
costs. 

Billing.  Effect  of  milestone  billing  ver¬ 
sus  cost  billing. 

Oversight.  Number  of  oversight  per¬ 
sonnel  per  program  budget  size. 

Cost  of  performance.  The  kind  of  .sys¬ 
tem  performance  that  can  be  bought  for  a 
given  cost.  To  derive  this  metric  it  would 


be  necessary  in  some  way  to  quantify  vari¬ 
ous  combinations  of  system  performance. 
This  is  a  formidable  ta.sk  open  to  contro¬ 
versy. 

Commercial  componentry.  Percent  of 
commercially  available  componentry: 
dollars  of  commercial  material  to  dollars 
of  total  obligation. 

System  gestation  time.  Time  fora  sys¬ 
tem  or  item  to  progress  from  concept  ex¬ 
ploration  and  definition  to  start  of  produc¬ 
tion  and  deployment  phase. 

Contractor’s  past  performance. 

Contractor’s  ranking  relative  to  other  con¬ 
tractors  on  a  predetermined  set  of  crite¬ 
ria. 

Government-unique  terms.  Propor¬ 
tion  of  government-unique  terms  and  con¬ 
ditions  to  total  number  of  such  tenns  and 
conditions  in  a  contract. 

Prote.sts.  Number  of  bid  prote.sts  per 
number  of  bidders. 

Regulatory  cost  premium.  DoD  cost 
premium  (%)  equals  contractor  compli¬ 
ance  costs  ($)  divided  by  value-added 
costs  ($)  X  100. 

Value-added  co.sts.  Value  added  costs 
as  percent  of  total  costs  where  value-added 
costs  equal  total  co.sts  minus  co.sts  of  ma¬ 
terial  purchases,  including  subcontracts 
minus  profit  minus  corporate  general  and 
admini.strative  allocations. 

Contractor  overhead.  Compare  per¬ 
centage  of  direct  and  indirect  costs  for  top 
defen.se  contractors  as  a  group  and  indi- 
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vidually  over  time.  Use  the  ratio  of  per¬ 
cent  indirect  costs  to  percent  direct  costs 
or  dollars  of  indirect  costs  to  dollars  of 
direct  costs. 

Consumable  item  price  index.  Cost  of 
a  Defense  Logistics  Agency  (DLA)  pre¬ 
determined  set  of  consumables. 

Contract  defaults.  Number  of  contract 
action  defaults  divided  by  the  total  num¬ 
ber  of  contract  actions. 

Contract  changes.  Number  of  contract 
changes  divided  by  the  total  number  of 
contracts. 

Contract  protests.  Number  of  protests 
resolved  using  the  alternative  dispute  reso¬ 
lution  process,  and  the  number  of  protests 
that  go  to  GAO  and  the  General  Service 
Board  of  Contract  Appeals  (GSBCA). 

Administrative  lead  time.  The  aver¬ 
age  time  from  the  signed  formal  require¬ 
ments  document  to  contract  award. 

Production  lead  time.  Time  from  con¬ 
tract  award  to  acceptance  of  first  item  or 
delivery. 

Engineering  changes.  Number  of  en¬ 
gineering  change  proposals  by  program 
phase  (demonstration  and  validation,  en¬ 
gineering  and  manufacturing  develop¬ 
ment,  production  startup). 

Alternative  specifications  and  stan¬ 
dards.  Number  of  contractors  offering  al¬ 
ternatives  to  military  specifications  and 
standards  per  100  proposals. 


Alternative  specifications  and  stan¬ 
dards  with  incentives.  Percentage  of  so¬ 
licitations  resulting  in  incentive  contracts 
where  alternatives  to  military  specifica¬ 
tions  and  standards  are  offered. 

Dissemination  time.  Time  for  process¬ 
ing  and  dissemination  of  requests  for  pro¬ 
posal,  statement  of  work,  and  specifica¬ 
tions  and  standards. 

Degree  of  use  of  simulation  and  mod¬ 
eling.  Percentage  of  contracts  over  $5 
million  using  simulation  and  modeling  to 
achieve  cost  performance  tradeoffs. 

Degree  of  activity-based  costing  and 
management.  Percentage  of  contracts  and 
contractors  that  use  activity-based  costing 
and  management.  (Activity-based  costing 
identifies  each  category  of  cost  [direct  or 
overhead]  and  relates  it  to  the  specific 
product  [e.g.,  military  specification  or 
standard,  statement  of  work  task,  etc.]  or 
product  line  that  causes  the  activity  to  be 
needed  and  performed.) 

Marginal  ownership  cost.  Cost  di¬ 
vided  by  operating  time. 

Technology  gestation.  Time  from  tech¬ 
nological  innovation  to  operational  sys¬ 
tem  integration. 

Cost  as  an  independent  variable.  Sav¬ 
ings  in  a  program  when  cost  is  used  as 
independent  variable. 

Operational  goals.  Probability  of 
achieving  or  exceeding  stated  operational 
profiles  in  a  specified  regime. 
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Reliability  goals.  Probability  of 
system’s  satisfactoiy  operation  in  given 
conditions. 

Maintenance  goals.  Proportion  of 
maintenance  activities  requiring  a  given 
level  of  maintenance. 

Integrability.  Ease  of  integrating  the 
new  system  into  an  existing  frame  or  or¬ 
ganizational  unit. 

Mean  time  between  failures  (MTBF). 

The  average  number  of  operating  hours 
between  system  failures. 

This  list  is  not  exhaustive.  Though  most 
of  these  metrics  have  been  taken  from 
DoD  sources,  it  is  not  clear  whether  all  of 
them  could  ser\’e  as  metrics  for  a  specific 
service’s  acquisition  process.  Still,  these 
examples  provide  some  insights  into  the 
types  of  metrics  that  are  being  considered 
and  the  levels  of  abstraction  that  are 
needed. 


Quality  FuNaioN  Deployment  and 
Analytic  Hierarchy  Process 


The  metrics  in  the  preceding  section  are 
clearly  at  different  levels  of  abstraction. 
For  instance,  “commercial  practice.s”  is 
more  abstract  than  “oversight,”  and  “in¬ 
tegrability”  is  more  abstract  than  “mean 
time  between  failures.”  Metrics  that  are 
not  very’  abstract  are  usually  easier  to  mea¬ 
sure.  Yet,  highly  abstract  metrics  are  of¬ 
ten  more  useful  for  as.sessing  a  proce.ss. 
We  propose  using  the  analytic  hierarchy 
process  (AHP)  and  quality  function  de¬ 
ployment  (QFD)  to  create  a  top-down  (or 


bottom-up)  hierarchy  of  metrics  that  trans¬ 
lates  what  we  can  measure  into  what  we 
are  interested  in  measuring. 

Using  QFD  and  AHP  to  Weight  Metrics 

QFD  is  a  structured  process  that  facili¬ 
tates  a  team  approach  to  identifying  and 
prioritizing  customer  requirements  and 
translating  these  requirements  into  appro¬ 
priate  company  requirements  at  each  stage 
of  the  product  life  cycle — from  research 
and  development  to  manufacturing  and 
support. 

QFD’s  stnictured  process  consists  of  a 
set  of  interrelated  matrices.  These  matri¬ 
ces  are  constructed  by  starting  with  the 
general  goals  (the  “whats”)  that  are  to  be 
achieved  and  then  selecting  the  various 
means  (the  “hows”)  for  achieving  those 
goals.  In  the  next  step,  the  current  “hows” 
become  the  goals  (i.e.,  the  “whats”)  and 
new  “hows”  are  identified  for  achieving 
the  new  “whats”.  This  process  is  repeated 
as  many  times  as  necessary  to  reach  a  de¬ 
sirable  level  of  detail. 

For  any  one  matrix,  the  elements  of  the 
matrix  are  intuitive  ratings  (using  the  scale 
weak  =  ],  medium  =  3,  high  =  9)  of  the 
contribution  of  a  column  means  (“hows”) 
to  a  row  goal  (“whats”).  The  relationship 
between  the  matrices  is  twofold:  the  col¬ 
umns  (HOWs)  of  a  matrix  become  the 
rows  (WHATs)  of  the  subsequent  matrix, 
and  the  computed  weights  of  a  matrix’s 
column  become  the  weights  of  the  subse¬ 
quent  matrix’s  rows.  Thus,  if  a  matrix  en¬ 
try  is  nr^  and  the  row  weights  are  w.  then 
the  computed  column  weights  are: 

L  w  m ,  /N,  X  w  m  JN,  X  w  m  JN, 

X  w  m  ,/N, .  . .  X  w  m,/N, 

^  I  i4  *  ^  i  ik  ’ 
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Figure  1.  Quality  Funclien  Deployment  (QFD)  for  Acquisition  Metrics 
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where  k  is  the  number  of  columns  and  N 
=  LZ  'V  m^Js  the  normalization  factor. 
These  weights  become  the  weights  of  the 
rows  for  the  subsequent  matrix.  Thu.s,  at 
any  given  step  the  analy.st  can  readily  a.s- 
certain  the  relative  contribution  of  a  means 
to  the  end  goals.  An  example  of  the  QFD 
process  for  metrics  is  shown  in  Figure  1. 

For  the  first  matrix,  the  row  weights 
could  be  derived  using  the  pairwise  com¬ 
parisons  of  the  AHP.  The  analytic  hierar¬ 
chy  process,  introduced  by  Saaty  in  1971, 
applies  a  structured  process  in  which  only 
two  factors  are  compared  at  a  time  (i.e., 
pairwise  comparisons )  to  a  complex  prob¬ 
lem  that  is  broken  up  into  manageable 
super  structures  forming  a  hierarchy 
(Saaty,  1980).  The  preferences  in  a 
pairwise  comparison  are  denoted  numeri¬ 
cally  by  a  scale  of  one  to  nine.  One  de¬ 
notes  equal  preference;  nine  denotes  ex¬ 
treme  preference.  The  mathematical  algo¬ 
rithm  of  AHP  converts  these  pairwise  pref¬ 
erences  into  rankings  of  relative  impor¬ 
tance  for  each  level.  AHP  provides  a 
framework  for  the  .selection  of  a  preferred 
alternative  in  a  context  of  conflicting  cri¬ 
teria.  For  the  mathematical  procedures  and 
details  of  the  process  .see  Saaty  (1980). 

Our  implementation  of  QFD  for  a  hier¬ 
archy  of  metrics  consists  of  the  following 
steps; 

Step  one:  Form  an  IPT.  Organize  an 
integrated  product  team  (IPT)  consisting 
of  members  that  adequately  represent  the 
concerned  community  with  re.spcct  to  the 
system  under  consideration. 

Step  two:  Construct  a  hierarchy  of 
metrics.  Charge  the  IPT  with  developing 
a  hierarchy  of  metrics.  For  instance,  the 
IPT  could  start  with  the  first  set  of  “whats” 


consisting  of  “better  system,”  “cheaper 
system,”  and  “faster  fielded  system.”  For 
the  “hows”,  the  IPT  could  choose  the  ar¬ 
eas  that  Preston  believes  warrants  the  defi¬ 
nition  of  metrics  and  some  areas  that  are 
of  importance  to  the  warfighter.  Thus,  the 
IPT’s  initial  set  of  “how.s”  could  consist 
of  cost,  acquisition  performance,  sched¬ 
ule,  commercial  practices,  system  perfor¬ 
mance,  and  technology  innovation. 

It  is  possible  that  an  IPT  would  come 
up  with  a  different  QFD  matrix.  However, 
we  believe  that  it  is  be.st  to  have  as  much 
commonality  as  possible  within  the  DoD 
community.  In  the  next  QFD  matrix  the 
“whats”  are  cost,  acquisition  performance, 
schedule,  commercial  practices,  system 
performance,  and  technology  innovation. 
See  Figure  I  for  an  example  of  a  hierar¬ 
chy  of  metrics  that  the  IPT  could  create. 

Step  three:  Derive  initial  weights.  As¬ 
sign  a  numeric  value  to  the  metrics  for 
“better  system,”  “cheaper  system,”  and 
“faster  fielded  system”  that  corresponds 
to  their  relative  importance.  In  this  ca.se 
the  IPT  must  decide  the  relative  impor¬ 
tance  of  only  three  metrics.  This  can  be 
done  directly  without  using  AHP.  How¬ 
ever,  each  IPT  member  should  perform 
this  assessment  individually  and  the  IPT 
should  average  the  values  using  the  geo¬ 
metric  mean. 

Step  four:  Fill  out  the  first  QFD  ma¬ 
trix.  Using  the  .scale  weak  =  1,  medium  = 
3,  high  =  9,  each  IPT  member  fills  out  the 
first  QFD  matrix  by  answering  the  ques¬ 
tion  “How  strongly  does  the  particular 
‘how’  reflect  the  particular  ‘what’?”  The 
IPT  average  matrix  entry  is  the  geometric 
mean  of  the  corresponding  individual  en¬ 
tries.  Note  that  we  assume  that  any  “how” 
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reflects  any  “what”  to  some  degree,  how¬ 
ever  weak.  Thus,  m^.  >  1. 

Step  five:  Compute  the  weights  of  the 
“hows.”  If  a  matrix  entry  is  m,,  and  the 
row  weights  are  w.,  then  the  computed  col¬ 
umn  weights  are: 

X  w.  m.,  /N,  S  w.  m ,/N, 

S  w.  m-j/N,  S  w.  m.^/N, 

. . .  X  w.  m  . /N, 

where  k  is  the  number  of  columns  and  N 
=  XX  w.  m,.  is  the  normalization  factor. 
These  weights  are  the  weights  of  the  rows 
for  the  subsequent  matrix. 

Step  six:  Repeat  this  process.  Repeat 
steps  four  and  five  as  many  times  as 
needed  to  develop  a  QFD  structure  that 
links  the  true  metrics  to  measurable  sur¬ 
rogate  metrics. 


Behehts  From  Applying  QFD _ 

Application  of  QFD  to  a  hierarchy  of 
metrics  can  provide  the  decision  maker 
with  significant  insights.  In  the  following 
we  list  some  of  these  insights  and  illus¬ 
trate  them  by  means  of  the  example  in 
Figure  1  (see  the  first  two  matrices).  Note 
that  the  entries  in  the  QFD  matrices  of  the 
figure  are  notional  and  that  the  rankings 
were  derived  using  the  QFD  algorithms. 

Application  of  QFD  to  a  hierarchy  of 
metrics  provides  indications  of  the 
metrics  on  which  to  concentrate  the 
data  collection  effort.  The  first  matrix  in 
Figure  1  ranks  the  “hows”  that  contribute 
to  the  true  metrics  that  measure  the  attain¬ 


ment  of  better  EW  systems,  cheaper  EW 
systenis,  and  faster  fielded  EW  systems. 
Metrics  that  pertain  to  schedule  and  com¬ 
mercial  practices  appear  to  have  been 
ranked  lowest  (.10).  This  could  indicate 
to  a  decision  maker  that  his  efforts  should, 
perhaps,  not  be  focused  on  these  metrics 
if  his  resources  are  limited.  The  decision 
maker  may  wish  to  defer  any  such  deci¬ 
sion  in  the  first  step  of  the  QFD  process 
and  wait  to  see  the  rankings  in  the  second 
step,  where  a  finer  substructure  is  as¬ 
sessed.  In  this  case,  they  will  find  that  such 
metrics  as  contractor  defaults,  administra¬ 
tive  lead  time,  and  logistic  response  time 
rank  very  low.  Now  the  decision  maker  is 
at  a  level  of  abstraction  and  detail  that 
permits  him  to  reconsider  the  collection 
of  data  for  the  measurable  surrogate 
metrics. 

Periodic  application  of  QFD  to  a  hi¬ 
erarchy  of  metrics  provides  a  means  for 
assessing  the  sensitivity  of  lower-level 
rankings  to  changes  in  higher-level 
rankings.  Suppose  that  the  DoD’s  empha¬ 
sis  has  changed.  Cheaper  EW  systems 
become  more  important  (e.g.,  .40)  and  the 
faster  fielding  EW  systems  become  less 
important  (e.g.,  .20),  while  better  EW  sys¬ 
tems  remained  as  important  as  before. 
How  would  this  change  effect  the  rank¬ 
ing  of  the  various  metrics?  The  decision 
maker  may  feel  that  this  change  in  em¬ 
phasis  is  considerable  and  would  affect  the 
rankings  of  the  metrics.  However,  the  re¬ 
calculation  of  “hows”  rankings  would 
show  that  they  did  not  change,  and  change 
in  current  practices  is  not  warranted. 

Suppose  that  there  is  another  change  in 
DoD’s  emphasis.  Cheaper  EW  systems 
become  more  important  (e.g.,  .45)  and  the 
faster  fielding  EW  systems  become  more 
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important  (e.g.,.45).  while  better  EW  sys¬ 
tems  become  less  important  (e.g.,  .1).  In 
this  case  the  importance  of  the  cost  metrics 
jumps  to  .39  while  that  of  system  perfor¬ 
mance  drops  to  .08.  Clearly,  the  decision 
makers  would  have  to  make  some  adjust¬ 
ments  and  shifts  in  their  efforts  if  they  are 
constrained  by  budgets. 


Application  of  QFD  to  a  hierarchy  of 
metrics  provides  a  means  for  monitor¬ 
ing  the  relative  importance  of  the 
metrics  as  a  function  of  time.  As  time 
passes,  the  “whats,”  the  “hows,”  or  both 

may  change.  If 

"If  the  QFD  cempux 
tfrtlont  are  repeirted 
at  fixed  intervals  of 
time  we  may  eb« 
serve  a  shift  in  the 
importance  of  vari> 
ous  metrics." 


the  QFD  com¬ 
putations  are  re¬ 
peated  at  fixed 
intervals  of 
time  we  may 
observe  a  shift 
in  the  impor¬ 


tance  of  various 
metrics.  Such  indications  could  provide 
the  decision  maker  with  the  necessary  time 
to  prepare  to  shift  from  one  set  of  metrics 
to  another. 


Application  of  QFD  to  a  hierarchy  of 
metrics  provides  a  means  for  compar¬ 
ing  the  metrics  for  the  acquisition  of  two 
distinct  weapon  systems.  Suppose  that 
the  QFD  matrices  in  Figure  1  were  ob¬ 
tained  for  an  airframe  and  a  similar  set  of 
matrices  (with  different  entries  and  initial 
weights)  was  obtained  for  the  avionics  of 
this  airframe.  The  differences  in  the 
rankings  could  then  provide  the  decision 
maker  with  interesting  information  on 
potential  problems  with  integration  of  the 
two  systems. 


Application  of  QFD  to  a  hierarchy  of 
metrics  provides  a  link  between  what 
we  can  measure  and  what  we  are  inter¬ 
ested  in  measuring.  Figure  1  indicates 
that  the  metric  “contract  changes”  (.12 
weight)  is  more  strongly  linked  to  the  true 
metrics  better  EW  systems,  cheaper  EW 
systems,  and  faster  fielded  EW  systems, 
than  the  metric  “logistic  respon.se  time” 
(.04)  weight. 


Understanding  the  Use  or  Metrics 
Statistical  Significance  of  Chance 

Suppose  that  we  have  identified  a  met¬ 
ric  for  a  ta.sk  or  a  process.  Next,  suppose 
that  this  metric  is  higher  for  the  current 
process  than  for  the  previous  process,  and 
higher  is  better.  Does  this  mean  that  the 
current  process  is  better  than  the  previous 
process?  Not  necessarily. 

First,  we  must  prove  that  the  difference 
is  stati.stically  significant.  However,  this 
requires  us  to  have  a  sizable  sample  of 
similar  cases,  make  assumptions  about  the 
population  probability  distribution,  and 
choo.se  the  statistics  that  will  be  used. 
Unfortunately,  this  kind  of  infonuation  is 
not  available  for  military-unique  pro¬ 
grams.  Consequently,  we  can  rarely  say 
with  certainty  that  a  positive  change  in  a 
metric  indicates  a  real  improvement.  We 
could  say  that  an  improvement  w'as 
achieved  only  if  the  change  in  the  metric 
was  spectacular. 

We  have  already  noted  that  many 
metrics  can  be  usually  defined  for  a  task 
or  process.  If  all  these  metrics  are  inde¬ 
pendent  and  point  in  the  positive  direc¬ 
tion,  then  we  would  be  more  certain  that  a 
positive  improvement  had  occurred.  On 
the  other  hand,  if  the  metrics  point  in  dif- 
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ferent  directions,  we  cannot  form  any  defi¬ 
nite  conclusion. 

Measurihg  Variations 

While  a  set  of  metrics  often  does  not 
allow  us  to  draw  solid  conclusions,  the 
comparative  metric  values  could  still  ben¬ 
efit  the  acquisition  process  by  indicating 
process  variations  that  cause  unsatisfac¬ 
tory  performance.  Thus,  linkages  between 
metrics  of  the  left  column  and  metrics  of 
the  right  column  in  Figure  2  could  serve 
as  indicators  of  process  variations.  How¬ 
ever,  such  linkages  can  be  established  with 
reasonable  confidence  only  if  data  from 
repeatable  processes  is  available. 

Repeatable  processes,  highly  desirable 
for  drawing  statistical  inferences,  are  usu¬ 
ally  unavailable  for  a  weapon  system  ac¬ 
quisition.  Such  an  acquisition  is  in  most 
cases  a  unique  event.  Moreover,  the  suc¬ 
cess  or  failure  of  the  acquisition  is  deter¬ 
mined  in  retrospect  by  how  well  the 
weapon  system  has  served  the  military. 
Consequently,  we  can  assess  the  success 


of  the  acquisition  process  only  in  a  post¬ 
mortem.  Our  challenge  is  to  define  metrics 
that  show  how  the  pieces  of  the  process 
are  doing  and  then  to  extrapolate  these  re¬ 
sults  to  the  entire  process. 

Conclusions 


We  have  outlined  requirements  and  pro¬ 
cedures  for  defining  meaningful  metrics 
for  the  acquisition  process.  It  does  not  pro¬ 
vide  a  prescription  for  the  generation  of 
metrics,  however.  For  some  processes 
useful  metrics  come  readily  to  mind,  for 
others  one  must  employ  substantial  in¬ 
sight  and  creativity.  But  in  each  case  the 
methods  presented  should  be  of  practical 
use. 

Since  directly  computable  metrics  tend 
to  be  limited  in  scope  and  specific  in  na¬ 
ture,  we  need  to  know  how  to  combine 
the  various  metrics  into  one  big  picture. 
We  propose  to  accomplish  this  by  appli¬ 
cation  of  QFD  or  AHR  In  these  processes 
we  do  not  actually  combine  the  metrics. 
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Figure  2.  Causes  of  Process  Variafien 
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but  rather  gauge  the  relative  contribution 
of  a  metric  at  one  level  to  the  metrics  at  a 
higher  level.  This  seems  to  be  a  more  pru¬ 
dent  path  to  follow,  because  past  efforts 
to  combine  metrics  numerically  have  usu¬ 
ally  failed. 

QFD  offers  intere.sting  opportunities  for 
linking  metrics  from  one  level  of  ab.strac- 
tion  to  a  higher  level.  The  ranking  of 
metrics  in  the  QFD  process  allows  one  to 
select  the  relatively  important  metrics. 
This  prioritization  could  lead  to  more  ef¬ 
ficient  strategies  for  assessing  the  acqui¬ 
sition  process. 


Definition  of  a  metric  is  the  beginning 
of  a  process  of  continual  refinement  for 
measuring  process  outcomes.  As  data  for 
a  metric  is  collected  and  the  metric  is  used, 
much  is  learned  that  could  shape  an  even 
better  metric.  Efforts  to  define  useful 
metrics  for  the  acquisition  reform  process 
must  focus  on  measures  that  give  insights 
into  reform  effects,  not  specific  acquisi¬ 
tion  program  indicators.  Employing  the 
methodology  outlined  in  this  paper  should 
help  to  keep  the  focus  where  it  ought  to 
be — on  reform  processes. 


ACKNOWLEDGMENT 

We  thank  Maj.  Art  Huber  for  his  constructive  remarks. 


204 


Selecting  Effective  Acquisition  Process  Metrics 


References 


Air  Force  System  Command.  (1991,  Au¬ 
gust).  The  Metrics  Handbook  (p.  3-3). 
Andrews  Air  Force  Base,  MD. 

Dellinger,  L.  (1994,  November-Decem- 
ber).  Secretary  Preston  addresses  DAU 
training  and  education  conference.  Pro¬ 
gram  Manager.  21. 

Fortuna,  R.  M.  (1988,  June).  Beyond  qual¬ 
ity:  Taking  SPC  upstream.  Quality 
Progress.  23. 


Meadows,  S.  I.  (1995,  July/August).  Army 
grapples  with  means  to  appraise  reform 
efforts.  National  Defense.  31. 

Office  of  the  Under  Secretary  of  Defense 
for  Acquisition  &  Technology.  (1994, 
April).  Report  of  the  Process  Action 
Team  on  Military  Specifications  and 
Standards.  Washington,  DC. 

Saaty,  T.  L.  (1980).  The  analytic  hierar¬ 
chy  process.  New  York:  McGraw-Hill. 


205 


Acquisifion  Review  Quarferly — Spring  1997 


Appendix  A 


This  list  of  strategic  outcome  metrics 
that  has  been  proposed  by  the  Defense 
Standards  Improvement  Council’s  Tiger 
Team. 

Cost.  Contractor  overhead,  cost  premium 
for  government  unique  requirements,  con¬ 
sumable  item  price  index,  government 
administrative  oversight,  military  speci¬ 
fications  conversion  price  benefit. 

Acquisition  performance.  Stability,  con¬ 
sumable  on-time  deliveries,  contract  pro¬ 
tests,  contract  defaults,  supplier  survey, 
contract  changes. 


Schedule.  Acquisition  phase  time,  ad¬ 
ministrative  lead  time,  production  lead 
time,  multiyear  procurements,  FACNET 
transactions,  logistics  response  time. 

Commercial  practices.  Commercial  con¬ 
tracts,  cost  and  pricing  data  reduction, 
commercial  content,  contract  specifica¬ 
tions,  commercial  market  share,  credit 
card  purchases. 
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Appendix  B 


Cost 


Consumable  item  price  index.  Total  cost 
to  DLA  customers  for  representative  sam¬ 
pling  of  consumable  items  expressed  in 
constant  dollars  (FY$). 

Military  specification  conversion  price 
benefit.  Cumulative  cost  avoidance  using 
commercial  specifications  (performance 
specs,  define  (NGS),  commercial  item  de¬ 
scription  (CIDs)  and  percent  cost  avoid¬ 
ance  each  year  for  Defense  Logistics 
Agency  (DLA)  order.  Applies  only  to 
items  converted  to  commercial  specifica¬ 
tions  since  previous  order. 

Acquisition  Performance 


Contract  defaults.  The  total  number  of 
contract  action  defaults  divided  by  the  to¬ 
tal  number  of  contract  actions. 

Contract  changes.  Waivers  and  devia¬ 
tions  of  major  significance  and  Class  I 
Engineering  Change  Proposals  (ECPs). 

Schedule 


Acquisition  phase  time.  Time  for  a  sys¬ 
tem  or  item  to  progress  from  concept  ex¬ 
ploration  and  definition  to  start  of  the  pro¬ 
duction  and  deployment  phase.  The  met¬ 
ric  is  the  average  number  of  days  between 
milestones  (e.g.,  number  of  days  between 
the  signatures  of  MS  0  and  MS  I). 


Administrative  lead  time.  The  average 
time  from  the  signed  formal  requirements 
document  to  contract  award.  This  excludes 
contracts  for  services  and  base  support. 

Multiyear  procurements.  The  number 
and  dollar  value  of  multiyear  procure¬ 
ments. 

Federal  Acquisition  Computer  Network 
(FACNET)  transactions.  The  dollar 
amount  of  FACNET  transactions  divided 
by  the  total  dollar  amount  of  transactions 
in  which  simplified  acquisition  procedures 
have  been  used;  the  number  of  FACNET 
transactions  divided  by  total  number  of 
transactions  in  which  simplified  acquisi¬ 
tion  procedures  have  been  used. 

Logistics  response  time.  The  time  be¬ 
tween  customer  order  and  customer  re¬ 
ceipt  of  DLA  items. 

Commercial  Practices 


Contract  specifications.  The  number  of 
specifications  and  standards  placed  on 
contracts  stratified  by  type:  performance 
specifications  (military  performance 
[MIL-PRF],  CIDs,  Product  Unique); 
NGS;  old  MIL-specs  (assumed  to  be  de¬ 
tailed);  and,  MIL-stds. 

Credit  card  purchases.  The  number  and 
dollar  value  of  credit  card  purchases. 
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TUTORIAL 


APPIYING  DIAIECTIC  TO 
ACQUISITION  STRATEGY 


David  L  Peeler,  Jr, 

Dialectic  is  the  process  of  reasoning  correctly.  In  the  era  of  downsizing  the 
defense  budget  and  streamlining  the  acquisition  process,  the  appiication  of 
dialectic  to  weapon  system  (acquisition)  programs  is  criticai.  Unless  programs 
are  debated  properly,  all  programs  and  defense  activities  stand  to  lose.  Dialectic 
operates  to  expose  the  best  in  each  program  idea,  creating  a  synthesis  that 
optimizes  the  selected  approach.  This  article  explains  the  concept  of  diaiectic 
and  how  its  use  can  improve  the  acquisition  process. 


Knowledge,  both  theoretical  and 
practical,  is  key  to  success.  Acqui¬ 
sition  strategy  formulation  is  the 
process  of  addressing  a  problem;  it  is  an 
event-driven,  iterative  process  that  at¬ 
tempts  to  answer  questions  that  involve 
the  possible  combinations  of  options  or 
approaches  available  to  achieve  a  system’s 
desired  objectives,  within  particular  limi¬ 
tations.  An  acquisition  strategy  attempts 
to  answer  the  why,  what,  when,  who,  and 
how  questions  that  have  to  do  with  ob¬ 
taining  a  system.  The  problem  inevitably 
has  many  potential  solutions.  The  dialec¬ 
tic  of  Kant,  Hegel,  and  others  provides  a 
framework  for  answering  the  aforemen¬ 
tioned  questions;  it  can  yield  better  insight 
into  the  convoluted  nature  of  the  acquisi¬ 
tion  strategy  process. 

The  decision  maker,  or  more  aptly  the 
decision  participant,  must  be  aware  of  the 
theoretical  underpinnings  of  the  debates 


in  which  he’s  engaged.  This  debate  be¬ 
tween  approaches  is  known  as  dialectic. 
A  theoretical  understanding  of  dialectic  by 
program  managers  (PMs)  is  crucial  to  the 
success  of  a  program.  The  decision  maker 
must  understand  the  interplay  and  integra¬ 
tion  within  a  program  and  between  it  and 
others.  Each  aspect  of  an  overall  acquisi¬ 
tion  strategy  is  subject  to  question.  There¬ 
fore,  the  PM  must  be  ready  to  defend  the 
program,  in  part  or  in  total,  in  order  to 
enhance  the  overall  welfare  of  Department 
of  Defense  (DoD)  acquisitions. 

This  goal  is  advanced  solely  through 
the  give  and  take  of  debate.  A  dialectical 
approach  yields  a  stronger  solution  to  the 
overarching  problem  (i.e.,  the  mission 
need  [the  why],  as  well  as  to  the  internal 
and  external  factors  (the  what,  when,  who, 
and  how).  Understanding  the  theoretical 
aspects  of  the  necessary  questions  and  is¬ 
sues,  whose  answers  are  integral  to  a  suc- 
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cessful  acquisition  strategy,  provides  a 
framework  for  improved  practical  appli¬ 
cations. 

The  first  theoretical  presentation  of  the 
dialectic  in  Western  thought  occurred  in 
Ancient  Greece.  Plato  writes  of  Socrates’s 
dialogue  with  Phaedrus  in  which  the  na¬ 
ture  or  essence  of  dialectic  is  examined. 
Although  the  notions  of  dialectic  can  be 
traced  further  back  through  Oriental 
works,  our  study  will  begin  with  its  ap¬ 
pearance  in  the  Occident.  Aristotle,  in  the 
Topics  (370  B.C./1987).  addresses  the  es¬ 
sence  and  existence  of  dialectic: 

A  dialectical  problem  is  a  subject  of 
inquiry  that  contributes  either  to 
choice  or  avoidance,  or  to  truth  and 
knowledge,  and  that  either  by  itself, 
or  as  a  help  to  the  solution  of  some 
other  such  problem.  It  must,  more¬ 
over,  be  something  on  which  either 
people  hold  no  opinion  either  way, 
or  the  masses  hold  a  contrary  opin¬ 
ion  to  the  philosophers,  or  the  phi¬ 
losophers  to  the  masses,  or  each  of 
them  among  themselves. 

Therein.  Aristotle  sets  the  stage  for  the 
modem.  Western  interpretation  of  dialec¬ 
tic;  it  is  a  problem  involving  opposing 
positions  concerning  the  solution,  either 
in  part  or  total.  As  for  the  differentiation 
between  the  masses  and  the  philosophers, 
recall  that  in  Ancient  Greece  the  Platonic 


hierarchy  established  philosophers  as  the 
be.st-suited  leaders — that  is,  the  decision 
makers.  This  hierarchy  was  used  through¬ 
out  the  literature  of  the  time. 

Socrates  further  stipulates  that  a  good 
dialectician  is  able  to  divide  things  by 
classes  or  subjects  “according  to  the  natu¬ 
ral  fomiation,  where  the  joint  is,  not  break¬ 
ing  any  part ....’’  (Plato,  350  B.C./1937). 
This  skill  requires  knowledge  of  the  thing 
in  question.  Thus,  dialectic  involves  a  con¬ 
flict  of  ideas  in  the  pursuit  of  a  solution  to 
a  problem — a  problem  that  must  be  famil¬ 
iar  to  the  dialectician. 

The  notion  of  dialectic  again  surfaces 
in  the  philosophy  of  Immanuel  Kant.  In 
his  Critique  of  Pure  Reason,  Kant  identi¬ 
fies  dialectic  as  a  part  of  the  hierarchy  of 
reason  (Figure  1 ). 


Figure  1.  Kant's  Hierarchy  off 
Reason 
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Kant’s  aesthetics  involves  the  senses 
while  logic  encompasses  intellect;  the  fur¬ 
ther  division  of  logic — into  analytic  and 
dialectic — is  concerned  with  understand¬ 
ing  and  refined  reasoning,  respectively 
(Bennett,  1974).  The  aesthetics  have  mini¬ 
mal,  direct  influence  on  the  dialectic  but 
play  a  role  in  the  overall  “scheme  of 
things.”  The  analytic  addresses  actually 
agreed  upon  “fact” — Karl  Popper  notwith¬ 
standing.'  For  example,  water  freezes  at 
32°F;  the  molecular  structure  of  water  is 
H^O.  Argument  about  these  “facts”  is 
vacuous  because  it  is  determined  by  ob¬ 
servation.  Dialectic  isn’t  established  by 
experiment  or  observation,  but  through  the 
negativism  of  ideas  (Weaver,  1953): 

. .  .we  can  therefore  say  that  a  dialec¬ 
tical  position  is  established  when  its 
relation  to  an  opposite  has  been  made 
clear  and  it  is  thus  rationally  rather 
than  empirically  sustained. 

The  actuality  is  not  what  “dialectic  se¬ 
cures  for  any  position. .  .but  possibility. . .” 
(Weaver,  1953).  The  analytic  represents 
the  actual;  this  is  determined  by  examina¬ 
tion.  The  dialectic  is  more  elusive;  while 
the  analytic  addresses  agreed-upon  facts, 
dialectic  deals  with  the  realm  of  interact¬ 
ing  possibilities  available  for  deriving  so¬ 
lutions  to  a  problem. 

Hegel  built  his  system  upon  Kant’s  con¬ 
cept.  Hegel  maintained  that  dialectic  has 
a  negative  character  which  “constitutes  the 
genuine  dialectical  procedure”  (Hegel, 
1929).  This  “...negativity  is  manifest  in 
the  very  process  of  reality,  so  that  nothing 
that  exists  is  true  in  its  given  form.  Every 
single  thing  has  to  evolve  new  conditions 
and  forms  if  it  is  to  fulfill  its  potentiali¬ 
ties”  (Marcuse,  1954).  The  negative  is 


manifested  as  the  antagonist  of  the  posi¬ 
tive  in  a  given  solution — the  positive  be¬ 
ing  the  existing  proposition  or  idea.  The 
negative  acts  as  the  devil’s  advocate  in  the 
proposed  solution.  The  notion  of  dialec¬ 
tic  therefore  cannot  be  put  in  a  single,  cor¬ 
rect  proposition  that  has  a  claim  to  the 
essence  of  the  problem  or  its  solution. 

No  one  approach  to  a  problem  exists, 
but  solutions  are  governed  by  the  creative 
power  of  contradiction.  These  contradic¬ 
tions  or  conflicts  of  ideas  are  intertwined 
in  the  process  of  reaching  a  solution.  The 
conflict  is  no  longer  between  opposing 
forces  but  becomes  one  between  antago¬ 
nistic  forms  of  reality  that  coexist 
(Marcuse,  1954).  Using  Hegel’s  concept 
of  triadic  development,  the  dialectic  forms 
the  essence  of  the  debate.  This  form  is  rep¬ 
resented  in  its  simplest  state  in  Figure  2. 


Figure  2.  Hegel's  Simple  Dialectic 


The  thesis  is  contradicted  (negated)  by 
an  antithesis  with  the  ensuing  conflict  be¬ 
tween  the  two  producing  a  higher  level 
(improved)  concept,  the  synthesis.  The 
positive  (thesis)  is  opposed  by  the  nega¬ 
tive  (antithesis),  producing  a  superior  syn¬ 
thesis.  The  best  of  both  the  positive  and 
the  negative  form  an  improved  idea.  In  a 
more  complicated  view,  the  synthesis  be¬ 
comes  the  thesis  and  the  iterative  process 
continues  to  evolve  (Figure  3).  The  pro- 
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Figure  3.  Mere  Complicated  View 
of  Hegel's  Dialectic 


cess  becomes  even  more  complex  with 
more  than  one  simultaneous  antitheses 
(Figure  4). 
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Figure  4.  Complex:  Simultaneous 
Antitheses 


The  evolution  of  this  process  is  relevant 
to  acquisition  strategy.  As  one  strategy 
component  is  proposed,  others  vie  for  pre¬ 
dominance.  The  conflict  of  ideas  through¬ 
out  the  acquisition  strategy  process  is  a 
dialectical  exercise. 

Practical  application  of  the  laws  of  rea¬ 
son  (dialectic)  are  inherent  in  the  evolu¬ 
tion  of  the  general  development  of  an  ac¬ 
quisition  strategy.  This  evolution  occurs 
through  an  iterative  process  to  create  a 
increasingly  effective  strategy.  The  itera¬ 
tive  incorporation  and  integration  of  co¬ 
existent,  alternative  approaches  produce 
a  new  synthesis.  Each  synthesis  is  met 
with  opposing  forces  that  perpetuate  fur¬ 


ther  syntheses,  thus  improving  the  acqui¬ 
sition  process. 

The  theoretical  laws  of  reason  as  sum¬ 
marized  above  are  applicable  to  all  aspects 
of  an  acquisition  strategy.  Dialectic  is 
present  in  the  give  and  take  between  DoD 
and  Congress.  “Every  new  solution  pre¬ 
sents  a  whole  new  set  of  problems” 
(L’Heureux  and  Grant,  1996).  Each  prob¬ 
lem  must  evolve  through  the  dialectical 
process  to  arrive  at  a  viable,  successful  ac¬ 
quisition. 

Acquisition  policy,  external  environ¬ 
ment,  and  program-specific  factors  are  the 
three  strata  of  acquisition  strategy  dia¬ 
grammed  in  the  text.  All  three  of  these 
make  u.se  of  dialectical  facets,  involving 
choices  that  must  be  broken  down,  as  ex- 
pres.sed  by  Socrates,  and  dealt  with  in  or¬ 
der  to  continuously  improve  the  outcome 
of  the  process.  Policy  is  established 
through  the  give  and  take  of  conflicting 
ideas.  Formulation  of  policy,  the  defini¬ 
tion  of  its  criteria,  and  the  development 
of  a  strategy  are  nothing  more  than  choices 
between  the  best  component  parts  articu¬ 
lated  by  the  participants.  Each  thesis  yields 
to  the  superior  ideas  included  in  its  an¬ 
tithesis.  The  re.sulting  synthesis  is  com- 
po.sed  of  the  best  of  the  opposing  postu¬ 
lates. 

Included  within  an  acquisition 
strategy’s  external  environment  are  the 
complex  aspects  of  congressional  over¬ 
sight,  exploitation  of  technology,  existence 
of  an  industrial  base,  joint  and  interna¬ 
tional  strategies,  and  strategies  for  com¬ 
petition  (L’Heureux  and  Grant,  1996).  All 
the.se,  save  the  latter,  are  subject  to  the 
same  coexistence  of  competing  ideas  that 
drive  the  resultant  solutions;  each  are 
made  up  of  conflicting  proposals  that  con¬ 
tain  good  and  less  than...  components. 
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The  best  aspects  are  taken  from  each  pro¬ 
posal.  Competition  is  less  a  factor  of  dia¬ 
lectic  and  more  of  an  analogous  concept. 
The  concept  of  Adam  Smith’s  “invisible 
hand”  refers  to  the  competition  between 
alternatives  with  the  result  being  a  better 
solution.  Economic  competition  thus  be¬ 
comes  somewhat  synonymous  with  dia¬ 
lectic;  they  have  the  same  essence.  How¬ 
ever,  this  essence  isn’t  as  easily  manifested 
in  major  weapon  system  or  automated  in¬ 
formation  system  acquisitions. 

The  essence  of  competition  involves  an 
environment  with  relatively  few  mles  and 
regulations;  easy  entry  and  exit  into  the 
marketplace,  a  large  number  of  firms,  a 
homogeneous  product,  and  complete  infor¬ 
mation  concerning  prices,  quality,  and  pro¬ 
duction.  The  market  surrounding  weapons 
system  (or  any  major  government  systems) 
acquisition  does  not  fit  the  competition 
model.  Our  marketplace  can  be  regarded 
as  a  monopsony  (one  buyer)  purchasing 
from  a  duopoly  (two  or  few  sellers).^ 
Therefore,  the  essence  of  dialectic  is  not 
obvious  in  acquisition  contract  competi¬ 
tion.  However,  dialectic  in  the  acquisition 
process,  from  conception  to  realization,  is 
essential  for  the  evolution  of  improved 
systems.  This  continuous  improvement  al¬ 
lows  for  the  optimization  of  resources  and 
mission  performance  capability. 

The  specific  factors  involved  in  an  ac¬ 
quisition  strategy  program  are  all  sub¬ 
ject  to  the  reasoning  process — the  dialec¬ 
tic.  These  factors  include  the  master  pro¬ 
gram;  contracting,  manufacturing,  sup- 
portability,  test  and  evaluation,  and  “high 
gear”  program  strategies;  commercial  off- 
the-shelf  (COTS)  and  nondevelopmental 
items  (NDI);  and  risk  assessment 
(L’Heureux  and  Grant,  1996).  Each  of 
these  factors  involve  the  application  of 


dialectic,  showing  that  the  laws  of  reason 
are  at  work  within  the  structure  of  an  ac¬ 
quisition  strategy  as  well  as  at  the  upper 
echelons  of  policy  formulation.  The  PM 
becomes  a  dialectician  on  two  levels;  He 
is  the  arbitrator  between  the  masses  within 
the  program,  determining  which  aspects 
of  competing  ideas  to  incorporate,  and  he 
becomes  the  program’s  advocate  vis-a-vis 
the  “philosophers”  or  between  them.  The 
PM  must  defend  the  program  as  it  is  placed 
in  competition  with  others.  He  must  be¬ 
come  an  advocate  in  order  to  ensure  that 
the  best  pieces  of  each  program  are  ad¬ 
vanced  and  synthesized  into  the 
overarching  framework  of  providing  the 
strongest  possible  national  security. 


Figure  5.  Program  Dialectic 


The  PM  must  be  cognizant  of  the  fact 
that  his  program  has  good  aspects  that 
improve  the  outcome  of  the  competition.^ 
If  he  immediately  sacrifices  his  program 
when  faced  with  resource  reductions,  the 
function  of  dialectic  is  frustrated.  Benefi¬ 
cial  ideas  will  be  lost  that  would  other¬ 
wise  survive  to  flourish  in  the  resultant, 
synthesis  program.  If  the  PM  doesn’t  fight 
for  the  program,  the  defense  of  the  nation 
suffers.  This  truism  is  likewise  applicable 
for  those  working  on  the  program.  They 
must  defend  their  ideas  for  the  program 
in  order  to  improve  it. 
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The  acquisition  strategy  process  is  a 
functionally  operating,  continuous  evolu¬ 
tion  of  thesis,  antithesis,  and  synthesis. 
Once  each  conflict  has  been  resolved  into 
a  synthesis,  a  new  antithesis  emerges.  So 
the  process  continuously  generates  im¬ 
proved  options  for  the  solution  of  a  prob¬ 
lem.  The  iterative  evolution  occurs 
through  dialectic.  The  dialectic  process 
places  potential  solutions  into  conflict, 


producing  an  improved  combination  of 
options  or  approaches  used  to  achieve  the 
desired  objectives  of  a  program  within 
specified  resource  constraints.  If  every  PM 
becomes  an  effective  and  active  dialecti¬ 
cian  rather  than  a  de  facto  one,  the  theo¬ 
retical  aspects  of  dialectics  will  improve 
the  practical  application  and  execution  of 
acquisition  strategy. 
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End  Notes 


1.  Karl  Popper  maintained  that  the  veri¬ 
fication  of  universal  scientific  state¬ 
ments  is  impossible.  Verification  would 
require  confirmation  of  all  instances 
through  time  and  universe.  The  scien¬ 
tific  method  cannot  accommodate  this 
verification  process;  therefore,  we  seek 
falsifying  instances  that  provide  one 
example  that  the  statement  is  not  uni¬ 
versal.  Kant’s  analytic  suggests  that  ob- 
sers’ation  connotes  fact.  Popper  would 
insist  that  simple  observation  does  not 
prove  scientific  universality  (i.e.,  the 
speed  of  light  in  a  vacuum  or  the  freez¬ 
ing  point  of  water 

2.  Although  the  evolving  acquisition  pro¬ 
cess  now  emphasizes  the  u.se  of  com¬ 
mercial  off-the-shelf  (COTS)  products, 
for  which  there  are  multiple  manufac¬ 
turers.  contracts  are  still  awarded  as  be¬ 
fore.  Lacking  the  necessary  expertise 
to  integrate  the  various  components 
that  make  up  a  system,  the  government 
contracts  with  one  firm  to  perform  this 
function.  The  number  of  firms  capable 
of  performing  the  integration  function 


is,  if  anything  (with  the  current  rate  of 
mergers  and  buyouts),  diminishing. 
Therefore,  the  monopsony/duopoly  re¬ 
lationship  between  the  government  and 
contractors  continues  to  hold  as  acqui¬ 
sition  methods  evolve.  COTS  products 
are  procured  through  prime  contrac¬ 
tors,  of  which  there  are  relatively  few 
for  each  acquisition  area,  who  integrate 
the  system  and  “load”  the  price. 

3.  The  natural  tendency  of  a  reader  is  to 
surmise  that  the  chosen  example  pro¬ 
gram  is  sufficiently  dissimilar  to  his  to 
exclude  u.se  of  the  example’s  processes. 
But,  the  dialectic  is  generic  and  applies 
to  all  processes  and  projects.  A  specific 
program  is  not  discussed  here  to  elimi¬ 
nate  the  tendency  of  readers  to  discount 
the  premises  because  this  or  that  ele¬ 
ment  wasn’t  or  isn’t  present  in  his  pro¬ 
gram.  The  process  of  debate  over  as¬ 
pects  present  in  any  program  is  what  a 
program  manager  needs  to  contem¬ 
plate,  not  the  analogous  nature  of  a 
particular  program  to  the  example. 
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COMMERCIAL  BEST  PRACTICES  AND 
THE  DOD  ACQUISITION  PROCESS 

James  5.  S.  Chew 


Continuous  improvement  continues  to  be  the  rallying  point  for  Department  of 
Defense  (DoD)  acquisition  reform.  The  recent  changes  to  the  DoD  5000  show 
that  the  department  is  streamlining  the  acquisition  process  to  meet  the  realities 
of  the  evolving  “new  world"  threats.  As  dramatic  as  the  changes  have  been, 
there  is  room  for  improvement.  Here  we  compare  the  streamlined  DoD 
acquisition  process  with  the  process  used  in  the  American  automobile 
industry— which  continually  deals  with  an  ever-evolving  threat.  We  discuss 
the  Chrysler  Corporation  product  development  process  and  identify  the  “best 
practices"  in  their  product  development  process.  These  best  practices  can  be 
applied  to  the  DoD  acquisition  process.’ 


The  basic  tenets  of  the  current  Depart¬ 
ment  of  Defense  (DoD)  acquisition 
reform  are  “better,  faster,  cheaper.” 
The  acquisition  reforms  have  resulted  in 
a  new  DoD  5000  mle  which  dictates  what 
should  be  performed  during  a  major  sys¬ 
tem  acquisition,  not  how  to  perform  one. 
Using  the  new  5000,  the  systems  devel¬ 
oped  should  be: 

1.  Better.  A  high-quality  system  must 
be  designed  and  built  right  the  first 
time. 

2.  Cheaper.  The  costs  of  developing, 
building,  fielding,  and  maintaining 
the  system  are  constrained  more  so 
than  in  the  past. 


3.  Faster.  The  streamlined  process  re¬ 
duces  the  amount  of  time  required 
to  acquire  and  properly  field  sys¬ 
tems. 

The  idea  of  DoD  acquisition  reform  is 
not  new.  Since  the  DoD  5000  was  first  is¬ 
sued  in  1971,  there  have  been  nine  revi¬ 
sions  in  an  effort  to  streamline  and  fine 
tune  the  DoD  acquisition  process 
(Ferrara,  1 996)  Table  1  presents  a  sum¬ 
mary  of  the  revisions  and  changes  to  the 
DoD  5000  since  it  was  issued. 

Even  with  the  latest  significant  acqui¬ 
sition  reforms,  there  is  room  for  improve¬ 
ment.  By  examining  a  commercial  prod¬ 
uct  development  process,  one  can  iden¬ 
tify  some  “best  practices”  to  further  im¬ 
prove  the  DoD  acquisition  process. 
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Commercial  Acquisition  Process 

Market  pressures  have  forced  U.S.  in¬ 
dustries  to  change  their  product  develop¬ 
ment  (system  acquisition)  process  to  pro¬ 
duce  products  in  a  “better,  faster,  cheaper” 
manner,  or  risk  extinction.  The  American 
automobile  industry-  is  a  good  example  be¬ 
cause  of  the  rapid  change  in  threat  (e.g., 
the  rise  of  the  Japane.se  automobile  indus¬ 
try-  in  the  early  1980s).  Chrysler  Corpora¬ 
tion.  in  particular,  evolved  unique  re¬ 
sponses  to  this  threat.  Forbes  magazine 
named  the  corporation  1997  “Company  of 
the  Year”  because  of  their  response  strat¬ 
egy  (Flint,  1977). 

Faced  with  the  possibility  of  bank¬ 
ruptcy  because  their  product  was  non¬ 
competitive.  Chry  sler  .studied  the  Japanese 
automakers  and  de\-eloped  their  own  prod¬ 
uct  development  process  to  significantly 
reduce  the  concept-to-production  timeline. 
In  addition  to  reducing  the  product  devel¬ 
opment  time  fron-i  60  to  30  months,  the 
product  requirements  process  was  refined 
to  ensure  that  the  customer  was  “de¬ 
lighted”  with  the  resulting  product  (Rou.sh, 
1996). 


The  Chrysler  Product  Development  and 
DoD  acquisition  process  timeline  are  com¬ 
pared  in  Figure  1 .  The  similarities  are  sur¬ 
prising,  with  the  one  exception — their  pro¬ 
gram  time  is  24  to  36  months  versus 
DoD's  7  to  12  years. 

Commercial  "Best  PRAaiCES": 

The  Chrysler  Model 


Chrysler  has  launched  about  30  new 
products  since  1991  using  the  process 
shown  in  Figure  1  (Chrysler,  1995).  Sales, 
market  share,  customer  satisfaction,  cus¬ 
tomer  loyalty,  corporate  profits,  and  dealer 
profits  have  all  significantly  increased 
during  that  time.  In  fact,  the  Chrysler  prod¬ 
uct  development  process  is  now  being 
studied  by  Japanese  and  European 
automakers. 

Key  elements  of  this  process  allowed 
Chiy'sler  to  achieve  its  goals  of  offering 
world-class,  leading-edge  products  in  a 
timely,  competitive  manner.  These  “best 
practices”  which  follow,  should  be  con¬ 
sidered  for  inclusion  into  any  future  DoD 
acquisition  reform  initiatives. 


Mr,  Chew  has  13  years  of  experience  in  rocket  propulsion,  space  system  and  weapons  tech¬ 
nology  development,  and  technology  acquisition.  He  holds  a  B.S.  degree  in  mechanical  engi¬ 
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tion,  he  is  a  product  marketing  consultant  for  Chrysler's  Dodge  Division,  and  is  an  adjunct 
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Figure  1.  Comparison  of  DoD  Acquisition  Timeline  to  Chrysler  Product 

Development  Timeline 
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Continuous  Improvement 

Although  the  Chrjsler  product  devel¬ 
opment  process  has  been  proven,  the  pro¬ 
cess  is  updated  and  refined  with  every  new 
product,  and  thus  incorporates  the  lessons 
learned  from  the  previous  development. 

Requirements  Benchmarking 

The  requirements  for  each  new  prod¬ 
uct  are  developed  through  aggressive 
benchmarking,  perfonned  both  inside  and 
outside  of  the  company.  In  external 
benchmarking,  prospective  customers  are 
survej’ed  to  detennine  what  they  want  in 
a  certain  product.  This  is  performed 
through  product  clinics  and  focus  groups: 
prospective  customers  are  brought  to  a 
central  location  and  surseyed  about  their 

likes,  dislikes, 
and  desires  con¬ 
cerning  certain 
competitive 
products.  They 
are  then  shown 
several  future 
product  con¬ 
cepts  to  deter¬ 
mine  what  fea¬ 
tures  should  be 
incorporated 
into  it.  The  ex¬ 
ternal  benchmarking  yields  a  rearview 
mirror  perspective  of  system  require¬ 
ments.  The  customers  can  only  tell  the 
designer  what  they  like  based  on  the  avail¬ 
able  choices. 

The  internal  benchmarking,  on  the  other 
hand,  is  performed  to  provide  the  vision 
of  what  the  future  products  should  be.  It 
is  performed  by  the  employees  studying 
and  testing  the  competition  to  determine 
the  “best  in  class”  attributes.  Based  on 
known  customer  desires,  quality  function 


deployment  exercises  to  detennine  the  key 
elements  of  customer  desires,  and  moni¬ 
toring  of  future  automotive  technology 
trends,  management  develops  projections 
of  what  these  attributes  would  most  likely 
be  by  the  time  the  developed  product  is 
launched  as  well  as  through  the  life  cycle 
of  that  product.  Through  aggressive 
benchmarking  and  continued  efforts  to 
reduce  product  development  cycle  time, 
Chrylser  is  able  to  challenge  yet  achiev¬ 
able  product  requirements  that  make  the 
developed  product  best  in  its  class  at 
launch  and  keep  it  competitive  until  a  new 
model  is  fielded. 

CROSS-FUNaiONAL  PLATFORM  TEAMS 

Chrysler  e.stabli.shed  product  develop¬ 
ment  “platform”  (e.g.,  small  car,  large  car, 
minivan,  Jeep-truck)  teams  that  incorpo¬ 
rate  all  the  di,sciplines  necessary  to  de¬ 
velop  a  product,  including  engineering, 
manufacturing,  sales,  marketing,  and  fi¬ 
nance.  The  platfonu  team  leader  is  given 
a  product  development  budget,  which  can¬ 
not  be  exceeded.  The  team  leader  is  al¬ 
lowed  to  shift  costs  when  the  team  feels  it 
is  appropriate,  but  the  product  develop¬ 
ment  schedule  must  be  maintained.  The 
platform  team  leader  uses  con.sensus  tech¬ 
niques  to  make  decisions  with  the  under¬ 
standing  that  once  decisions  are  made,  the 
entire  platform  team  supports  them.  Past 
performance  has  shown  that  the  cross¬ 
functional  process,  team  member  empow¬ 
erment,  and  the  desire  for  continuous  im¬ 
provement  have  reduced  the  development 
cycle  time  for  each  new  product. 

Independent,  Aggressive  Advanced 
Concept  Development 

To  meet  future  product  requirements 
and  market  conditions,  Chrysler  estab- 


"Past  performance 
has  shewn  that  the 
cross'functlenal 
process,  team  mem« 
her  empowerment, 
and  the  desire  for 
continuous  Improve¬ 
ment  have  reduced 
the  development 
cycle  time  for  each 
new  product/' 
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lished  an  independent  advanced  product 
development  platform  team  that  develops 
conceptual  vehicles  that  meet  time-phased 
technology  goals  five  to  ten  years  into  the 
future.  During  the  development  of  these 
advanced  concept  vehicles,  the  engineers 
and  designers  identify  technologies,  pro¬ 
cesses,  and  components  that  require  de¬ 
velopment.  The  advanced  concept  engi¬ 
neers  and  designers  write  technical  memo¬ 
randums  to  the  other  platform  develop¬ 
ment  teams  that  outline  (a)  the  technol¬ 
ogy,  component,  or  process  they  have  de¬ 
veloped;  (b)  when  it  will  be  ready  for  in¬ 
corporation  into  a  production  vehicle;  (c) 
the  competitive  advantage  of  the  technol¬ 
ogy,  component,  or  process;  and  (d)  tech¬ 
niques  for  incorporating  that  technology, 
process,  or  component. 

This  advanced  concept  development 
team,  known  as  Chrysler  Liberty,  is  lo¬ 
cated  separate  from  the  rest  of  the  plat¬ 
form  development  teams.  Customer  reac¬ 
tion  to  these  technologies  and  design  con¬ 
cepts  is  gauged  by  showcasing  advanced 
concept  vehicles  at  the  annual  major  auto 
shows  (Moore,  1997). 

Program  Stability 

At  the  concept  approval  phase,  the  cor¬ 
porate  officers  and  the  platform  team 
leader  agree  to  a  “contract”  in  which  the 
corporate  officers  approve  a  product  de¬ 
velopment  budget  and  schedule  and  the 
platform  team  leader  agrees  to  produce, 
field,  and  establish  the  required  logistics 
and  operational  support  for  a  product  that 
meets  the  established  requirements.  Dur¬ 
ing  product  development,  the  budget  re¬ 
mains  stable  and  the  product  requirements 
are  changed  only  if  the  competition  sur¬ 
prises  the  platform  development  team. 
However,  all  changes  are  performed  by  the 


team.  The  corporate  officers  are  briefed 
on  the  progress  of  the  work. 

Addressing  Product  Quality 
Upfront  and  Early 

The  automobile  manufacturers  and 
their  supplier  base  are  striving  to  work 
toward  the  ISO  9000  quality  and  reliabil¬ 
ity  standard.  However,  ISO  9000  speci¬ 
fies  what  is  expected  of  a  quality  system, 
not  “how”  to  establish  a  quality  system. 
Figure  2  presents  the  detailed  tasks  that 
dictate  the  how  for  the  concept  phase  of 
the  Chrysler  product  development  process 
(Roush,  1996).  This  phase  takes  place 
prior  to  concept  approval.  Note  the  num¬ 
ber  of  quality  and  reliability  tasks  that  are 
performed  during  this  phase.  The  number 
of  quality  and  reliability  tasks,  including 
product  serviceability  and  assembly,  sig¬ 
nificantly  increase  with  each  product  de¬ 
velopment  phase.  These  quality  and  reli¬ 
ability  tasks  are  required  of  Chryler’s  sup¬ 
plier  base  also  (Lesniack,  1996).  Suppli¬ 
ers  are  graded  on  their  component  quality 
and  reliability;  Chrysler  does  help  suppli¬ 
ers  that  are  having  problems  in  these  ar¬ 
eas. 

Because  an  assembly  line  shutdown  for 
any  reason  costs  the  manufacturer  ap¬ 
proximately  $3,000  a  minute,  it  behooves 
both  the  manufacturer  and  the  supplier  to 
ensure  that  quality  components  are  deliv¬ 
ered  to  that  line  on  time.  Chrysler  has 
found  that  “design  for  manufacturing”  is¬ 
sues,  such  as  design  of  experiments  to 
identify  the  manufacturing  variables  and 
manufacturing  lessons  learned  from  pre¬ 
vious  products,  must  be  identified  and 
addressed  during  the  concept  development 
phase.  This  avoids  the  use  of  components 
or  assembly  procedures  that  require 
unique  processes.  Not  only  does  this  help 
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ensure  high  product  and  component  qual¬ 
ity,  but  ultimately  lowers  the  cost  of  prod¬ 
uct  development  by  eliminating  costly  re¬ 
designs  and  manufacturing  processes.  By 
addressing  quality  and  reliability  aggres¬ 
sively  and  early  in  product  development, 
the  need  for  material  review  boards  to  ad¬ 
dress  noncompliant  component  issues  are 
significantly  reduced. 

Applying  Commercial  "Best 
Practices"  To  DoD  Acquisition 

Applying  some  or  all  of  these  commer¬ 
cial  best  practices  to  the  DoD  acquisition 
process  would  decrease  the  acquisition 
costs  and  timeline  as  well  as  significantly 
improve  the  quality  and  reliability  of  the 
delivered  system.  But,  each  of  these  prac¬ 
tices  has  an  impact;  we  will  now  discuss 
some  of  them. 

Continuous  Improvement 

The  basis  for  following  the  Chrysler 
product  development  continuous  improve¬ 
ment  philosophy  is  the  establishment  of  a 
process  baseline.  This  baseline  should 
identify  and  discuss  in  detail  the  signifi¬ 
cant  tasks  that 
must  be  per¬ 
formed  during 
each  pha.se.The 
expected  “de¬ 
liverables”  for 
each  milestone 
should  also  be 
identified  and 
discussed.  The  tasks  and  deliverables  may 
be  modified,  eliminated,  or  added  as  a  re¬ 
sult  of  “lessons  learned.”  Using  such  a 
manual,  with  lessons  learned  from  previ¬ 


ous  programs  incorporated,  program  man¬ 
agers  for  new  acquisition  category  (AC AT 
I)  programs,  such  as  the  Joint  Air-to-Sur- 
face  Standoff  Missile  (JASSM),  the  Joint 
Strike  Fighter  (JSF),  the  Evolved  Expend¬ 
able  Launch  Vehicle  (EELV),  and  the  Sur¬ 
face  Combatant-21  (SC-21),  would  de¬ 
velop  an  initial  program  acquisition  flow¬ 
chart,  with  detailed  tasks.  The  govern¬ 
ment-industry  team  would  modify  or  re¬ 
vise  them  as  the  programs  progress  though 
Milestone  III,  and  the  charts  would  be¬ 
come  the  basis  for  an  ACAT  1  program 
acquisition  manual.  Subsequent  ACAT  1 
programs  would  modify  the  manual  as 
their  “lessons  learned”  accrued. 

Requirements  Benchmarking 

The  two  major  parties  that  need  to  work 
closely  with  the  weapons  designers  dur¬ 
ing  the  requirements  benchmarking  pro¬ 
cess  are  the  product  customer  (i.e.  the 
warfighter,  which  includes  the  operations, 
maintenance,  and  logistics  communities, 
and  major  operational  commands)  and  the 
DoD  science  and  technology  community. 
The  product  customer  helps  the  designer 
identify  the  deficiencies  with  the  current 
systems  and  the  needs  that  they  would  like 
a  new  system  to  fulfill.  The  DoD  science 
and  technology  community,  through  a  fo¬ 
cused,  time-phased,  goal-oriented  pro¬ 
gram,  helps  the  designer  identify  the  level 
of  technology  that  will  be  available  for  the 
proposed  new  system.  By  combining  the 
customer  comments  and  the  science  and 
technology  available,  prototype  systems 
on  either  the  component,  subcomponent, 
or  “virtual”  level  would  be  developed  and 
used  for  customer  product  “clinics.”  The 
weapons  designer  would  use  the  data  and 
information  from  both  groups  and  the  clin- 


"Too  often  in  the 
post,  coordination 
with  the  science  and 
technology  commu¬ 
nity  did  not  occur 
until  well  after 
Milestone  0." 
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Figure  2,  Product  Assurance  Planning:  Concept  Phase 
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ics  to  project  what  requirements  the  new 
system  would  need  in  order  to  be  a  “bench¬ 
mark”  system  when  fielded,  and  continue 
to  be  extremely  competitive  through  its 
life  cycle.  Once  these  system  requirements 
are  set.  they  should  not  be  changed  or 
modified  during  the  program  acquisition. 
This  step  needs  to  be  performed  prior  to 
Milestone  0.  Too  often  in  the  past,  coor¬ 
dination  with  the  science  and  technology 
community  did  not  occur  until  well  after 
Milestone  0. 

CROSS-FUNaiOMAL  PLATFORM  TEAMS 

As  is  the  Integrated  Product  Team 
(IPT),  the  cross-functional  platform  team 
would  be  developed  “upfront  and  early” 
in  the  program  (prior  to  Milestone  0).  As 
with  Chry  sler,  the  relevant  industrial  base 
would  be  brought  in  to  provide  a  reali.stic 
determination  of  technology  readiness  and 
co.sts.  These  teams  would  be  re.sponsible 
for  either  developing  or  modifying  their 
program  acquisition  flowchart  and 
manual. 

Independent,  Aggressive  Advanced 
Concept  Development 

To  provide  focus  for  the  DoD  science 
and  technology  community  and  provide 
ideas  for  future  weapon  systems  design¬ 
ers  and  customers,  more  advanced  con¬ 
cept  demonstrations  would  need  to  be  con¬ 
ducted  as  part  of  the  DoD  science  and 
technology  program.  In  addition  to  pro¬ 
viding /orn.s  to  the  DoD  labs  and  the  in¬ 
dustrial  base,  these  hardware  demonstra¬ 
tions  provide  a  mechanism  to  address 
quality  and  reliability  is.sues  of  incorpo¬ 
rating  the  new  technology  “upfront  and 
early,”  and  provide  an  opportunity  for  the 


weapons  designer  to  address  technology 
transition  and  integration  issues. 

The  DoD  science  and  technology  pro¬ 
gram  would  become  even  more  efficient 
through  these  technology  demonstrations, 
providing  the  program  had  time-phased 
technology  goals  that  would  be  used  con¬ 
sistently  for  the.se  demonstrations.  Exist¬ 
ing  DoD  science  and  technology  projects 
such  as  the  Integrated  High  Performance 
Turbine  Engine  Technology  (IHPTET) 
program  and  the  Integrated  High  Payoff 
Rocket  Propulsion  Technology  (IHPRPT) 
program  have  already  developed  time- 
phased  technology  goals,  which  will  be 
demonstrated  to  help  provide  the  weap¬ 
ons  designer  with  propulsion  system  de¬ 
sign  options  for  future  systems. 

Program  Stability 

Program  stability  requires  both  require¬ 
ments  stability,  which  should  be  addressed 
through  aggressive  requirements 
benchmarking,  and  resource  allocation 
process  reform  (Planning,  Programming, 
Budget,  Scheduling  and  Congressional 
Budget  enactment)  (Moore,  1997).  The  fo¬ 
cus  of  this  reform  effort  would  be  to  not 
require  annual  acquisition  program  justi¬ 
fications,  and  to  provide  the  acquisition 
program  manager  with  the  entire  required 
program  budget  at  Milestone  1  and  allow 
the  program  manager  to  manage  that  bud¬ 
get  through  the  course  of  that  program. 
Concepts  such  as  two-year  appropriations 
and  multi-year  procurement  would  help 
provide  program  stability.  The  ability  to 
commit  future  Congressional  appropria¬ 
tions  as  well  as  tnist  in  the  acquisition  pro¬ 
gram  manager  is  necessary  to  provide  the 
necessary  program  stability. 
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Addressing  Product  Quality 
Up-front  and  Early 

Most  of  the  DoD  AC  AT  1  programs  are 
addressing  the  “cost  of  quality”  issue 
(Lesniack,  1996).  The  cost  savings  pro¬ 
jected  by  some  of  these  programs,  the 
Evolved  Expendable  Launch  Vehicle 
(EELV)  in  particular,  are  from  significant 
reductions  in  the  cost  of  quality.  The  EELV 
program  is  using  a  “no-MRB”  (Materials 
Review  Board)  strategy,  which  puts  the 
quality  burden  primarily  on  the  contrac¬ 
tor.  This  implies  applying  the  various  qual¬ 
ity  design  techniques  during  the  concept 
exploration  phase,  which  requires  more 
funding  at  the  beginning  of  the  program. 
The  acquisition  program  does  not  require 
more  funding  overall,  just  more  up  front. 
In  fact,  the  Japanese  have  shown  that  ap¬ 
plying  more  funding  at  the  beginning  of  a 
product  development  program  can  actu¬ 
ally  reduce  program  costs  and  schedule. 
This  is  very  much  in  keeping  with  the 
maxim,  “an  ounce  of  prevention  is  worth 
a  pound  of  cure.”  The  major  impact  is  that 
instead  of  funding  several  small  system 
development  concept  contracts  with  “seed 
money,”  the  DoD  and  industry  will  need 
to  do  more  work  together,  through  con¬ 
tractual  vehicles,  to  include  more  manu¬ 
facturing  and  technology  risk  reduction 
during  Phase  0.  This  will  be  most  effec¬ 
tive  when  combined  with  aggressive  re¬ 
quirements  benchmarking  and  program 
stability. 


Summary 


It  should  be  noted  that  these  “best  prac¬ 
tices”  used  by  the  Chrysler  Corporation 
were  developed  as  a  result  of  the  Chrysler 
Corporate  Officers  realization  that  the  way 
they  were  conducting  business  would  re¬ 
sult  in  their  bankruptcy.  A  corporate  cul¬ 
ture  to  “reinvent”  the  company  to  be  the 
best  automaker  in  the  world  was  “brought 
in”  by  the  entire  company,  and  the  prod¬ 
uct  development  process  they  now  use  is 
a  result  of  this  effort.  By  using  their  pro¬ 
cess,  Chrysler  has  not  only  reduced  their 
product  development  time  from  60  to  24 
months,  but  they  have  significantly  re¬ 
duced  product  development  costs.  Key  to 
this  transition  was  the  willingness  of  the 
company  to  experience  short  term  set¬ 
backs  during  the  “re-invention  period”  for 
the  significant  long  term  gains. 

The  DoD  is  in  a  similar  situation — the 
way  we  are  doing  business  does  not  fit 
with  the  new  world  realities.  While  the 
current  DoD  Acquisition  Reforms  have 
been  applauded  by  industry  leaders,  there 
is  an  acknowledgment  that  they  do  not  go 
far  enough  (Augustine,  1996).  Shorter  sys¬ 
tem  development  cycle  times,  aggressive 
future  system  benchmarking  initiatives, 
and  focused,  aggressive  advanced  concept 
development  efforts  are  practices  which 
would  serve  to  keep  DoD  systems  and  the 
industrial  base  on  the  “leading  edge.”  Al¬ 
though  applying  the  aforementioned 
commerical  “best  practices”  may  not  be 
“the”  acquisition  reform  answer  that  many 
in  the  defense  industry  desire,  it  will  go  a 
long  way  toward  meeting  that  goal. 
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Table  1.  The  500  Series  Historical  Perspective  (continued) 
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Table  1.  The  500  Series  Historical  Perspective  (continued) 
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ACQUISITION  REVIEW  QUARTERLY 
GUIDELINES  FOR  CONTRIBUTORS 


The  Acquisition  Review  Quarterly 
(ARQ)  is  a  scholarly  peer-reviewed  jour¬ 
nal  published  by  the  Defense  Acquisition 
University.  All  submissions  receive  a 
masked  review  to  ensure  impartial  evalu¬ 
ation. 


Submissions 


Submissions  are  welcomed  from  any¬ 
one  involved  in  the  Defense  acquisition 
process.  Defense  acquisition  is  defined  as 
the  conceptualization,  initiation,  design, 
development,  test,  contracting,  produc¬ 
tion,  deployment,  logistic  support,  modi¬ 
fication,  and  disposal  of  weapons  and 
other  systems,  supplies,  or  services  to  sat¬ 
isfy  Defense  Department  needs,  or  in¬ 
tended  for  use  in  support  of  military  mis¬ 
sions. 


Research  Articles 


Manuscripts  should  reflect  research  or 
empirically-supported  experience  in  one 
or  more  of  the  aforementioned  areas  of 
acquisition.  Research  or  tutorial  articles 
should  not  exceed  4,500  words.  Opinion 
pieces  should  be  limited  to  1,500  words. 

We  publish  Defense  Acquisition  re¬ 
search  articles  that  involve  systemic  in¬ 


quiry  into  a  significant  research  question. 
The  article  must  produce  a  new  or  revised 
theory  of  interest  to  the  acquisition  com¬ 
munity.  You  must  use  a  reliable,  valid  in¬ 
strument  to  provide  your  measured  out¬ 
comes. 


Manuscript  SEaiONS 


The  introduction  should  state  the  pur¬ 
pose  of  the  article  and  concisely  summa¬ 
rize  the  rationale  for  the  undertaking. 

The  methods  section  should  include  a 
detailed  methodology  that  clearly  de¬ 
scribes  work  performed.  Although  it  is 
appropriate  to  refer  to  previous  publica¬ 
tions  in  this  section,  the  author  should  pro¬ 
vide  enough  information  so  that  the  expe¬ 
rienced  reader  need  not  read  earlier  works 
to  gain  understanding  of  the  methodology. 

The  results  section  should  concisely 
summarize  findings  of  the  research  and 
follow  the  train  of  thought  established  in 
the  methods  section.  This  section  should 
not  refer  to  previous  publications,  but 
should  be  devoted  solely  to  the  current 
findings  of  the  author. 

The  discussion  section  should  empha¬ 
size  the  major  findings  of  the  study  and 
its  significance.  Information  presented  in 
the  aforementioned  sections  should  not  be 
repeated. 
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Research  Considerations 


Contributors  should  also  consider  the 
following  questions  in  reviewing  their  re¬ 
search-based  articles  prior  to  submission: 

•  Is  the  research  question  significant? 

•  Are  research  instruments  reliable  and 
valid? 

•  Are  outcomes  measured  in  a  way 
clearly  related  to  the  variables  under 
study? 

•  Does  the  research  design  fully  and  un¬ 
ambiguously  test  the  hypothesis? 

•  Did  you  build  needed  controls  into  the 
study? 

Contributors  of  research-based  submis¬ 
sions  are  also  reminded  they  should  share 
any  materials  and  methodology  necessary 
to  verify  their  conclusions. 

CRinRiA  FOR  Tutorials 


Tutorials  should  provide  special  in¬ 
struction  or  knowledge  relevant  to  an  area 
of  defense  acquisition  to  inform  the  De¬ 
fense  Acquisition  Workforce. 

Topics  for  submissions  should  rely  on 
or  be  derived  from  observation  or  experi¬ 
ment,  rather  than  theor)'.  The  submission 
should  provide  knowledge  in  a  particular 
area  for  a  particular  purpose. 


Opinion  Criteria 


Opinion  articles  should  reflect  judg¬ 
ments  based  on  the  special  knowledge  of 
the  expert.  Opinion  articles  should  be 
based  on  observable  phenomena  and  pre¬ 
sented  in  a  factual  manner;  that  is,  sub¬ 
missions  should  imply  detachment.  The 
ob.servation  and  Judgment  should  not  re¬ 
flect  the  author’s  personal  feelings  or 
thoughts.  Nevertheless,  opinion  pieces 
should  clearly  express  a  fresh  point  of 
view,  rather  than  negatively  criticize  the 
view  of  another  previous  author. 

Manuscript  Style 


We  will  require  you  to  recast  your  last 
version  of  the  manu.script,  especially  ci¬ 
tations  (e.g.,  footnotes  or  endnote.s)  into 
the  fonnat  required  in  two  specific  style 
manuals.  The  ARQ  follows  the  author 
(date)  form  of  citation.  We  expect  you  to 
use  the  Publication  Manual  of  the  Ameri¬ 
can  Psychological  Association  (4th  Edi¬ 
tion),  and  the  Chicago  Manual  of  Style 
(14th  Edition).  The  ARQ  follows  the  au¬ 
thor  (date)  form  of  citation. 

Contributors  are  encouraged  to  seek  the 
advice  of  a  reference  librarian  in  complet¬ 
ing  citations  of  government  documents. 
Standard  formulas  of  citations  may  give 
only  incomplete  information  in  reference 
to  government  works.  Helpful  guidance 
is  also  available  in  Gamer,  D.L.  and  Smith, 
D.H.,  1993,  The  Complete  Guide  to  Cit¬ 
ing  Government  Documents:  A  Manual 
for  Writers  and  Librarians  (Rev.  Ed.), 
Bethe.sda,  MD:  Congressional  Informa¬ 
tion  Service,  Inc. 
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Copyright  Information 


The  ARQ  is  a  publication  of  the  United 
States  Government  and  as  such  is  not 
copyrighted.  Contributors  of  copyrighted 
works  and  copyright  holders  of  works  for 
hire  are  strongly  encouraged  to  request 
that  a  copyright  notification  be  placed  on 
their  published  work  as  a  safeguard  against 
unintentional  infringement.  The  work  of 
federal  employees  undertaken  as  part  of 
their  official  duties  is  not  subject  to  copy¬ 
right. 

In  citing  the  work  of  others,  it  is  the 
contributor’s  responsibility  to  obtain  per¬ 
mission  from  a  copyright  holder  if  the  pro¬ 
posed  use  exceeds  the  fair  use  provisions 
of  the  law  (see  U.S.  Government  Printing 
Office,  1994,  Circular  92:  Copyright  Law 
of  the  United  States  of  America,  p.  15, 
Washington,  DC:  Author).  Contributors 
will  be  required  to  submit  a  copy  of  the 
written  permission  to  the  editor  before 
publication. 


Manuscript  Format 


Pages  should  be  double-spaced  and  or¬ 
ganized  in  the  following  order:  title  page, 
abstract,  body,  reference  list,  author’s  note 
(if  any),  and  figures  or  tables.  To  ensure 
anonymity,  each  paper  should  be  submit¬ 
ted  with  a  separate  page  that  includes  the 
author(s)’s  name(s)  and  complete  address, 
and  the  paper  should  include  the  title,  ab¬ 
stract,  keywords,  body,  complete  set  of 
references,  along  with  tables  and  figures 
at  the  end.  Authors  are  reminded  not  to 
refer  to  themselves  or  to  their  own  work 
directly  in  the  paper.  Figures  or  tables 
should  not  be  inserted  (or  embedded,  etc.) 


into  the  text,  but  segregated  one  to  a  page 
following  the  text.  Articles  must  be  print¬ 
able  within  one  issue  and  should  not  ex¬ 
ceed  4,500  words  for  research  or  tutorials 
and  1,500  words  for  opinion  pieces;  ar¬ 
ticles  will  not  be  printed  in  parts  or  in  a 
continuing  series.  If  material  is  submitted 
on  a  computer  diskette,  each  figure  or  table 
should  be  recorded  in  a  separate,  export¬ 
able  file  (i.e.,  a  readable  .eps  file).  For 
additional  information  on  the  preparation 
of  figures  or  tables,  see  CBE  Scientific 
Illustration  Committee,  1988,  Illustrating 
Science:  Standards  for  Publication, 
Bethesda,  MD:  Council  of  Biology  Edi¬ 
tors,  Inc.  Please  restmcture  briefing  charts 
and  slides  to  a  look  similar  to  those  in  pre¬ 
vious  issues  of  ARQ. 

The  author  (or  corresponding  author  in 
the  case  of  multiple  authorship)  should 
attach  to  the  manuscript  a  signed  cover 
letter  that  provides  the  author’s  name,  ad¬ 
dress,  and  telephone  number  (fax  and 
Internet  addresses  are  also  appreciated). 
The  letter  should  verify  that  the  submis¬ 
sion  is  an  original  product  of  the  author; 
that  it  has  not  been  published  before;  and 
that  it  is  not  under  consideration  by  an¬ 
other  publication.  Details  about  the  manu¬ 
script  should  also  be  included  in  this  let¬ 
ter:  for  example,  its  title,  word  length,  the 
need  for  copyright  notification,  the  iden¬ 
tification  of  copyrighted  material  for 
which  permission  must  be  obtained,  a  de¬ 
scription  of  the  computer  application  pro¬ 
grams  and  file  names  used  on  enclosed 
diskettes,  etc. 

The  letter,  one  copy  of  the  printed 
manuscript,  and  any  diskettes  should  be 
sturdily  packaged  and  mailed  to:  Defense 
Systems  Management  College,  Attn: 
DSMC  Press  (ARQ),  9820  Belvoir  Road, 
Suite  G38,  Fort  Belvoir,  VA  22060-5565. 
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In  most  cases,  the  author  will  be  noti¬ 
fied  that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following 
an  initial  review,  submissions  will  be  re¬ 
ferred  to  referees  and  subsequent  consid¬ 
eration  by  the  ARQ  Editorial  Board. 


Contributors  may  direct  their  questions 
to  the  Editor,  ARQ,  at  the  address  shown 
above,  by  calling  (703)  805-4290  (fax 
805-  2917),  or  via  the  Internet  at 
gonzalezd@d.smc.dsm.mil. 

The  DSMC  Home  Page  can  be  accessed 
at; 

http://www.dsmc.dsm.mil. 
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LEHER  TO  THE  EDITOR 


Having  recently  read  the  Winter  1997 
(Vol.  1 ,  No.  4)  edition  of  the  Acquisition 
Review  Quarterly,  I  wanted  to  write  a  letter 
to  the  authors  of  the  article  entitled  “Rethink¬ 
ing  Twenty-first  Century  Acquisition: 
Emerging  Trends  For  Efficiency  Ends.” 

The  Department  of  Defense  has  oper¬ 
ated  various  arrangements  with  industry 
for  support  and  services  for  years.  To  make 
the  topic  of  the  so  called  “...three  emerg¬ 
ing  public-sector  trends — privatization, 
promotion  of  competition,  and  continuous 
process  improvement...”  appear  to  be  in 
the  vanguard  of  the  twenty-first  century 
is  wrong. 

The  authors  do  not  recognize  nor  dis¬ 
cuss  the  fundamental  question  that  each 
acquisition  program  must  handle  and 
make  a  decision:  What  should  be  the  best 
support  structure  for  the  life  of  the  equip¬ 
ment/program.  It  is  not:  What  is  the  cheap¬ 
est,  or  how  does  WAL-MART  do  it?  When 
was  the  last  time  WAL-MART  deployed? 
The  glib  use  of  metrics  from  the  private 
sector  may  sound  good,  but  the  Depart¬ 
ment  of  Defense  does  not  have  stockowner 
value  as  its  goal — it  has  the  often  fuzzy, 
always  debatable  concept  of  readiness  to 
fight  and  win  as  its  goal. 

The  decision  to  use  a  particular  support 
structure  for  a  given  program  has  been 
based  on  such  elements  as  lack  of  adequate 
resources  in  the  program  to  political  con¬ 
siderations  at  the  local  level.  For  the  Pro¬ 
gram  Manager  or  Acquisition  Executive 
the  “policy”  decision  is  the  degree  of  or¬ 
ganic  versus  non-organic  support.  The  fac¬ 
tors  impacting  that  policy  decision  range 
from  economic  to  political,  but  if  that  ini¬ 


tial  policy  decision  isn’t  made,  then  the 
program  flounders  in  different  uncoordi¬ 
nated  approaches  and  ends  up  costing 
much  more  than  it  should. 

This  is  what  is  happening  today.  Be¬ 
cause  there  is  a  lack  of  a  disciplined,  quan- 
,  titative  approach  to  the  support  stmcture 
policy  decision,  programs  are  placed  un¬ 
der  pressure  to  carve  out  areas  for 
privatization  and  unequal  competition 
between  public  and  private  sectors.  The 
pressure  to  go  “private”  will  increase  as 
corporations  see  the  market  availability’s 
shift  from  end  items  to  life  cycle  support 
and  maintenance.  There  has  already  been 
a  shift  away  from  competition  among  sup¬ 
pliers  as  original  equipment  manufactur¬ 
ers  try  to  retain  their  market  share  and  have 
stepped  up  their  proprietary  claims. 

The  conclusion  of  the  authors,  that  the 
“...policy  initiatives  are  sweeping  public 
sector  management...”  and  that  “...pro¬ 
gram  managers. ..must  implement  policies 
to  improve  efficiency  or  face  the  threat  of 
privatization  themselves”  is  the  wrong 
signal  and  an  erroneous  conclusion.  The 
acquisition  community  experts  should  be 
exploring  methodologies  to  evaluate  these 
different  support  structures,  to  take  the 
discussion  out  of  the  political  arena  and 
into  the  unemotional  field  of  quantitative 
analysis,  to  develop  the  metrics  for  de¬ 
termining  the  degree  of  organic  versus 
nonorganic  supports  and  to  provide  the  de¬ 
cision  maker  with  information  needed  to 
make  informed,  useful  business  decisions. 

CAPTA.  E.  Steigelman,  SC,  USN 

Arlington,  VA 
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AUTHOR  RESPONSE  TO 
lEHER  TO  THE  EDITOR 


We  believe  intellectual  interchange  is 
exactly  what  the  Acquisition  Review 
Quarterly  is  all  about.  Discussion,  debate, 
and  deliberation  in  the  arena  of  ideas  is 
welcomed  to  help  advance  general  knowl¬ 
edge  and  in  this  case,  our  understanding 
and  appreciation  of  acquisition  in  the  21.st 
century'.  Offering  our  article,  “Rethink¬ 
ing  21st  Century  Acquisition;  Emerging 
Trends  for  Efficiency  End.s"  to  the  acqui¬ 
sition  community  provides  just  one  per¬ 
spective  of  what  the  future  of  acquisition 
may  look  like.  We  are  enthused  and  thank 
Captain  A.E.  Steigelman  for  his  comments 
and  insights  on  our  article. 

Continuing  the  spirit  and  inquiry  of 
Capt  Steigelman’s  letter,  we  do  however 
disagree  with  some  of  his  remarks.  First 
and  foremost,  our  fundamental  thesis  is 
that  as  we  know  it  today  program  manag¬ 
ers  and  the  major  weapon  .system  acquisi¬ 
tion  community  are  not  immune  from  the 
possible  elimination  and  replacement  by 
the  private  sector.  As  such,  we  suggest 
the  acquisition  program  manager  should 
not  only  be  aware  of  other  efficiency  im¬ 
provement  initiatives  occurring  in  the  fed¬ 
eral  government  but  be  implementing 
them  when  possible  where  they  make 
sense.  This  proposition  is  based  on  the 
current  wave  of  apparently  more  efficient 
alternative  approaches  proposed  by  Con¬ 
gress.  many  Federal  agencies,  as  well  as 
the  Executive  branch.  Tlie  countless,  well- 
documented  instances  of  privatization, 
outsourcing,  and  0MB  Circular  A-76 
competitions  literally  choke  the  popular 


press,  calling  for  the  government  to  inves¬ 
tigate  alternative  means  to  perform  diverse 
functions  such  as  the  space  program,  de¬ 
pot  maintenance,  and  ba.se  facility  man¬ 
agement.  Is  acquisition  management  so 
peculiar  from  these? 

Secondly,  our  article  does  not  focus  on 
“what  is  cheapest”  or  “how  Wal-Mart  does 
it”  as  Capt  Steigelman  implies.  In  fact  we 
agree  with  Capt  Steigelman  that  there  is 
an  increasing  emphasis  on  life  cycle  op¬ 
tions  and  a  search  for  metrics.  We  pro¬ 
pose  financial  analysis  and  Activity  Based 
Co.sting  (ABC)  as  potential  alternatives  to 
measuring  efficiency  and  cost  basis,  not 
as  panaceas. 

Third,  we  find  it  unrealistic  to  “...take 
the  discussion  out  of  the  political  arena 
and  into  the  unemotional  field  of  quanti¬ 
tative  analysis...”  as  Capt  Steigelman  sug- 
ge.sts.  Those  of  us  in  the  acquisition  com¬ 
munity  do  not  have  the  luxury  of  ignoring 
the  political  environment,  nor  what  other 
federal  agencies  are  doing.  As  we  reflect 
on  the  National  Performance  Review,  the 
Commission  on  Role  and  Missions,  the 
Defense  Science  Board,  the  GAO  and  many 
other  entities,  all  are  prodding  the  DOD  to 
outsource  and  privatize  more,  not  less. 

Finally,  regarding  his  que.stion,  “When 
was  the  last  time  Wal-Mart  deployed?”, 
we  offer  another  question,  “When  was  the 
last  time  Brown  &  Root  Service  Corpora¬ 
tion  deployed?”  One  finds  the  deployment 
history  of  this  private  sector  business  to 
include  Somalia,  Rwanda,  Haiti,  South¬ 
west  Asia,  Bosnia  and  Croatia  (AFJ,  Oct 
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96  pp.  19-22).  Granted  we  recognize  cer¬ 
tain  functions  will  always  be  “inherently 
governmental”  in  nature.  It  is  clear  how¬ 
ever  if  front  line  units  are  outsourcing  and 
privatizing,  the  acquisition  community 
should  be  aware  of  and  investigating  these 
emerging  trends  for  efficiency  ends. 

Capt  Steigelman’s  letter  alludes  to  some 
very  well-thought  and  documentable  ideas 
and  suggestions  which  we  enjoyed  read¬ 
ing.  We  suggest  he  continue  with  this  line 
of  thought  and  provide  these  suggestions 


to  the  acquisition  community  to  build  on 
the  energy  generated  from  this  article  and 
our  dialogue.  As  this  discussion  shows,  we 
all  firmly  believe  we  are  in  this  together. 
The  only  people  that  are  wrong  in  such  a 
debate  are  people  that  think  the  system  is 
working  at  peak  efficiency  and  promote 
the  status  quo. 

Conrad  S.  Ciccotello,  Major,  USAF 
Steve  G.  Green,  LTC,  USAF 
Martin  J.  Homyak,  Major,  USAF 


The  ARQ  welcomes  Letters  to  the  Editor,  but  retains  the  right  to  edit 
letters.  All  attempts  will  be  made  to  honor  original  content  and  context 
as  space  permits.  Send  letters  to: 

Defense  Systems  Management  College 
ATTN:  Debbie  Gonzalez 
9820  Belvoir  Road,  Ste.  G38 
Fort  Belvoir,  VA  22060-5565 

or  e-mail  to: 

gonzalezd  @  dsmc  .dsm.mil. 
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The  ARQ  would  like  to  clarify  an  error 
made  to  the  article  entitled  “A  Holistic 
Management  Framework  for  Software 
Acquisition,”  by  Dr.  Yacov  Haimes,  Dr. 
Richard  Schoof,  and  Mr.  Clyde  Chittister. 
Dr.  Schoof  and  Dr.  Haimes’  names  were 
inadvertently  switched.  Dr.  Schoof  is  the 
senior  author,  followed  by  Dr.  Haimes  and 
Mr.  Chittister. 


We  would  like  to  apologize  to  the  au¬ 
thors  of  the  article  entitled  “Development 
of  Emerging  Telemedicine  Technologies 
within  the  Department  of  Defense.”  We 
inadvertently  dropped  Dean  Calcagni’s 
name  as  an  author  on  the  article’s  title  page 
and  omitted  Timothy  Cramer’s  biography. 
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Call  for  Papers 

Acquisition  Review  Quarterly 

—  Special  Issue  on  Managing  Radical  Change  — 

The  Acquisition  Review  Quarterly  (ARQ)  The  focus  of  this  special  issue  is  on  man- 
is  planning  a  special  issue  with  the  theme  aging  radical  change,  which  has  roots  in 
“Managing  Radical  Change.”  We  are  the  literature  on  business  process 
pleased  to  issue  this  call  for  papers!  reengineering,  change  management,  new 
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